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in this issue ... 



Boston in December was a hot place to 
be. So thought the attendees of the 
Twelfth Systems Administration 
Conference, LISA '98. You missed it? 
Read on: Rik Farrow muses on the con¬ 
ference, Tina Darmohray draws a moral 
from it, and the Conference Reports 
offer other perspectives. 

Something new in the SAGE section: Hal 
Pomeranz introduces our “How-to Notes” series with a clear and concise set of instruc¬ 
tions for installing anonymous FTP. Among other offerings, John Sellens goes to the 
heart of reliability issues: the users themselves - how to communicate with them, how 
to enlist their assistance, and how to keep them informed. And Hal Miller considers the 
increasingly complex issues of system security. 


In the Features section, Dario Forte continues the discussion of security with a survey 
of current intrusion-detection systems: how effective are they in monitoring and 
reporting network attacks? Focusing on Squid as a firewall component, Jeffrey Mogul 
reports on his experience tracking down and eliminating a major inefficiency in Squids 
CPU utilization. On a lighter note, Spike Ilacqua Yesses up to his role in perpetrating 
the first ISP. 


Before you dive in, try your hand at our latest acquisition, the USENIX Crossword 
Puzzle. 
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letter to the editor 


On Free Software 

[Editors Note: I received this as private 
email from Matty who is a freshman at 
MIT. He consented to publishing it.] 

Dear Rob: 

On reading the latest Microsoft trial news 
and all, I keep hearing about how Linux 
is so great. I went over to <www.slashdot.org>, 
a site that appears to be for and by open- 
source software (OSS) fanatics, and the 
things they say absolutely disgust me. 

It was always my impression that most 
Linux fans simply disliked Microsoft. 
That’s okay. But what’s not okay, in my 
mind, is that they seem to go much fur¬ 
ther than that. They don’t just dislike 
Microsoft. I may be generalizing unfairly, 
but it seems that these OSS fanatics are 
opposed not only to specific companies’ 
products but to the entire idea of capital¬ 
ism. 

They keep referring to how OSS repre¬ 
sents freedom, how they are paving the 
way for a world in which capitalism will 
become obsolete. One went so far as to 
say that capitalism would be dead in 50 
years simply due to OSS ideology. To me, 
OSS looks like a step backward, not a step 
forward. These OSS tenets seem just like 
those of communism all over again, just 
under a different, more lucrative name 
and a new breed of supporters. Freedom? 
Yeah, the freedom to work really hard on 
something and get no rewards beyond 
hacker pride and the idea that you’ve 
contributed to a “good cause.” 

I was always under the impression that 
OSS fans liked Netscape. Well, obviously 
not. They were attacking the new 
Netscape/AOL/Sun alliance as if it was 
Microsoft. It’s not Microsoft that they 
target, it’s all corporations. This, despite 
Netscape having actually released its 
source code for its browser. 


One person referred to a future in which 
people wouldn’t work for money but for 
love and fame and such things. Well, if 
that’s the future, and that’s their idea of 
progress, you can count me out of that 
future! Working for love and fame is only 
going to work for about a week before 
you realize that you can’t feed or clothe 
yourself and sure as hell can’t live the 
“good life”! Certainly no gigantic houses 
with racquetball courts, swimming pools, 
or 10-foot widescreen TVs! 

I’m tired of OSS fans; maybe they should 
win, just so they can see how bad their 
future is. In particular, since a lot of them 
work day jobs and do OSS as a hobby, I 
think it would amusing to watch them 
drive their own companies out of busi¬ 
ness with their free software and then 
have to beg for change on the streets: 
“Starving Linux developer. Will work for 
food.” 

Remember the Halloween memo? They 
all treat it as if Microsoft was being 
unreasonable in being scared of OSS, that 
it was proof of their theory that corpora¬ 
tions are inherently oppressive of “free¬ 
dom.” Well, maybe it’s just me, but I 
would have thought that there would be a 
lot of people out there who would com¬ 
pletely empathize with Microsoft. People 
ask how they can compete against 
Microsoft, but competing against 
Microsoft is incomparable to competing 
with the Linus Torvalds of the world, 
who (1) don’t want to make a profit from 
their work and (2) don’t have to pay their 
“employees.” If I were Linus Torvalds, I’d 
be hitting myself really hard for somehow 
managing to make a product used by 
millions of people and yet somehow hav¬ 
ing managed to not get rich off of it. 

Did I mention that it appears they want 
to make intellectual property illegal (or 
at least some of them do)? I bet they 
wouldn’t be so enthusiastic about a 
change like that if they actually thought it 
through and realized that it would mean 


that there would be zero incentive to cre¬ 
ate intellectual property, since everyone 
would just rip it off. First effect, no more 
media companies. TV would die. Radio 
would die. The commercial parts of the 
Internet would die. Many books would 
never be published. Then no more soft¬ 
ware companies. What company would 
want to create a software program they 
had no rights to? Red Hat and others 
have a working business model simply 
because they don’t have to create Linux. 
Others do it for them, and they sell sup¬ 
port. But if Red Hat had to develop Linux 
itself, they’d be bankrupt in six months. 

In other words, their prescription for 
freedom of information would certainly 
free information to be used by all, but it 
would also backfire and make it so that 
no one would ever want to generate new 
information! 

Making this all the more real and fright¬ 
ening to me is the fact that MIT is the 
place of origin of OSS, and that its stu¬ 
dents seem to be the biggest proponents 
of it around. There are a horrifyingly 
large number of Linux fanatics among 
the students here. 

If this is the way our economy is going in 
the future, count me out of it. Give me 
plain old capitalism any day. 

Matthew Craighead 
<craighea@MIT.EDU> 
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LETTER TO THE EDITOR 





conference 

reports 


This issue’s reports focus on the 
Twelfth Systems Administration 
Conference (LISA ’98), held in 
Boston, MA, on December 6-11, 
1998. 

Our thanks to the summarizers: 

Chastity D. Arthur ■■■■■■ 

Kurt Dillard 
Carolyn M. Hennings 
Brian Kirouac 
Douglas Stewart 
Allen Supynuk 
John Talbot 
Bruce Alan Wynn 


KEYNOTE ADDRESS 

The Evolution of Open Source Software 

Eric Allman, Sendmail, Inc. 

Summary by Carolyn M. Hennings 



Eric Allman 


The original author of sendmail, Eric 
Allman, has been actively involved in all 
aspects of software development. He pro¬ 
vided an authoritative view of the history, 
current state, and possible future of what 
we now call open-source software. 

Allman reminded us that the first imple¬ 
mentations of computer systems were 
mainframes that required special facilities 
and staff. These systems came with the 
source code for the operating system, and 
all configuration information was in the 
source code. The next systems to become 
available were minicomputers designed 
originally for lab environments. These 
systems were less expensive and required 
less support, but the users were more 
sophisticated. Some commercial software 
started coming on the scene, but most 
software was free, and the source code 
was available. With both the mainframe 
and minicomputer systems, the hardware 
blueprints were available. 

The advent of microcomputers changed 
the characteristics of the computer-user 


community. Microcomputers were 
affordable and available to the hobbyist. 
As the hardware became more and more 
available and software became easier and 
easier to use, the demand for systems 
increased. With the availability of cheap, 
reliable hardware, the need for blueprints 
and specifications decreased. At the same 
time, the demand and need for knowl¬ 
edge about the internal workings of the 
computer software decreased. These fac¬ 
tors led us to our current situation of 
cheap hardware and expensive software. 

The recent movement toward open- 
source software has been built on the 
desire and contributions of avid technol¬ 
ogists who were frustrated by the inabili¬ 
ty to access, understand, and optimize the 
software. The expansion of the Internet 
and the resulting ability quickly and easi¬ 
ly to share code has provided a forum for 
this community to develop software on 
“Internet time” 

Allman referred to “The Cathedral and 
The Bazaar” article <http://www.tuxedo.org/ 
~esr/writings/cathedral-bazaar/> by Eric S. 
Raymond comparing the commercial 
centralized software-development process 
to a cathedral and the decentralized 
development of open-source software on 
the Internet to a bazaar. Allman suggested 
that a moderator is necessary for truly 
successful open-source software develop¬ 
ment. The “bazaar” analogy does not 
extend to this level. He described a num¬ 
ber of new company models supporting 
the open source philosophy in different 
ways: 

■ Give away old source code, sell new. 

■ Develop and give away source code, sell 
service. 

■ Market and give away source code, sell 
service. 

■ Develop and give away base set of 
source code, sell ease-of-use extensions, 
other tools, and service. 

■ Develop and give away parts, sell most 
as closed source. 
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■ Develop and give away source, place 
restrictions on the subsequent use of 
the code. 

Allman’s answer to the question, “Why 
give it away?” is that “religious” argu¬ 
ments don’t make good business. 
Sendmail, Inc. continues giving away the 
source code in order to maintain market 
share, ensure universal interoperability, 
and improve the quality of the code. He 
commented that the volunteer individu¬ 
als likely to support and contribute to 
open-source code development are natu¬ 
rally focused on the “interesting” and 
“cool” work. However, commercially 
viable software requires development that 
might be less exciting, such as GUIs for 
configuration and/or management tools. 
His proposal is to develop and sell these 
extensions as a commercial package, 
while continuing to release the base soft¬ 
ware in the open-source arena. 

Allman predicted that features-based 
code will tend toward open source, while 
algorithm-based code will generally stay 
closed. He also suggested that open- 
protocol-based code will tend toward 
open source and that, subsequently, mis¬ 
sion-critical applications will tend toward 
open protocols. 

REFEREED PAPERS TRACK 

Session: Security 

Summary by Kurt Dillard 

TITAN 

Dan Farmer, Earthlink Network; Brad 
Powell, Sun Microsystems, Inc.; 
Matthew Archibald, KLA-Tencor 

Titan is a freely available host-based tool 
that can be used to audit or improve the 
security of a UNIX system. It started as a 
Bourne Shell script to reconfigure various 
daemons. Checks for verifying configura¬ 
tions were added, and over time Titan 
became an effective tool for auditing 
computers. The authors made it clear 
that this is a powerful tool not designed 


for the weak or timid sysadmin. Using it 
incorrectly, you could easily render a sys¬ 
tem unusable or even unbootable. For the 
SA willing to put in the time to learn 
Titan thoroughly, it can save a great deal 
of time while helping to verify and main¬ 
tain security across multiple hosts. The 
authors also made it clear that Titan is 
not the be-all and end-all of informa¬ 
tion-systems security; it is designed to be 
only part of the overall infrastructure. 
Titan now runs on most versions of 
Solaris, but it shouldn’t be too difficult to 
port the scripts to other flavors of UNIX. 
By editing the scripts you can reconfigure 
Titan so that it performs auditing and 
configuration changes appropriate to the 
type of host you are running it on and 
the security policies that your network 
requires. 

See <http://www.fish.com/security/titan.html>. 

Infrastructure: A Prerequisite for 
Effective Security 

Bill Fithen, Steve Kalinowski, Jeff 
Carpenter, and Jed Pickel, CERT 
Coordination Center 

The authors started their presentation 
with some scary data compiled by CERT. 
A 1997 survey shows that 50% of systems 
were not kept up to date with security 
patches after they were compromised. 

One site appeared in 35 incidents 
between 1997 and 1998; the site was used 
for password sniffing and probing of 
other sites in many of those cases. Ten of 
the 35 incidents involved root compro¬ 
mise of the host. In another break-in, 20- 
25 hosts were compromised. All of these 
systems needed to be rebuilt, but the site’s 
administrator said that they didn’t have 
enough resources to do so. The authors 
set out to improve infrastructure man¬ 
ageability at CERT by creating an easily 
maintained system of distributing soft¬ 
ware packages. The result is SAFARI, a 
centralized repository of 900 collections 
of software for multiple versions of 
UNIX. Using SAFARI, a sysadmin can 


build new systems from scratch and 
update existing systems with patches and 
new packages. SAFARI includes flexible 
version controls so that developers and 
admins can easily post and retrieve soft¬ 
ware packages from the same central 
repository. 

SSU: Extending SSH for Secure Root 
Administration 

Christopher Thorpe, Yahoo!, Inc. 

Christopher Thorpe needed to create a 
low-cost method of allowing distributed 
access to privileged operations. As a sys¬ 
tem administrator at Harvard University, 
he had 200 systems to manage, and many 
of the students, staff, and faculty required 
root access to one or more processes on 
those systems. A method was needed to 
allow these users to execute certain 
processes as root in a secure environ¬ 
ment. SSU combines SSH and Perl to cre¬ 
ate a system that allows this by combin¬ 
ing RSA key pairs with those commands. 
Unfortunately, users need a separate key 
for each command that they need access 
to, but the solution works at Harvard 
because most of the users need to execute 
only a few commands with root access. A 
side-benefit of SSU is that everyone is 
now using SSH for all of their console 
connections, making all network activity 
more secure. 

Session: Pushing Users and 
Scripts Around 

Summary by Allen Supynuk 

System Management with NetScript 

Apratim Purakayastha and Ajay 
Mohindra, IBM T.J. Watson Research 
Center 

NetScript is a BASIC-like (could also be 
Perl- or Tcl-like) scripting language for 
remote administration of heterogeneous 
systems (UNIX, Wintel, and soon PDAs 
like the PalmPilot). It includes neat fea¬ 
tures like parallel scripts, isolation, and 
disconnected operation. 
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Single Sign-On and the System 
Administrator 

Michael Fleming Grubb and Rob 
Carter, Duke University 

This presentation did a nice job of cover¬ 
ing the major interpretations and issues 
involved in single sign-on, but, not sur¬ 
prisingly, was a bit short on satisfying 
solutions. The paper is worth reading for 
anyone interested in this holiest of grails. 

Session: Storage Performance 

Summary by Carolyn M. Hennings 

Using Gigabit Ethernet to Backup Six 
Terabytes 

W. Curtis Preston, Collective 
Technologies 

Curtis Preston presented his paper and 
some interesting additional information. 
He talked about two different backup sys¬ 
tems he implemented and some of the 
things he learned along the way. One 
conclusion from these experiences was 
that the limitation in backing up this 
amount of data in a reasonable time- 
frame was really in the network. He sug¬ 
gested that private “storage-area net¬ 
works” will be the future direction. 

Configuring Database Systems 

Christopher R. Page, Millennium 
Pharmaceuticals _ 

Christopher Page described relational 
databases from a system-management 
viewpoint. Relational databases reside on 
computer systems that are managed by 
system administrators. The system 
administrator needs to be knowledgeable 
about how the database works with, and 
uses, the operating system. Page began 
with information about the relational- 
database architecture. From a database 
user s perspective, data is conceptualized 
in tables; it is accessed and controlled 
through the Structured Query Language 
(SQL), and data manipulation is transac¬ 
tion-oriented. From the relational-data¬ 


base server perspective, it is an operating 
system on top of the native operating sys¬ 
tem, it manages the steps in processing 
requests, and it maintains four different 
file types (data, log, temporary, and con¬ 
trol files). Issues that system administra¬ 
tors need to be aware of and work with 
the database administrators on include: 
setting up and monitoring memory 
structures; optimizing “Intimate Shared 
Memory” (ISM); and configuring swap 
space, CPU, and network usage. Page 
noted that some key things to be aware of 
are data-block size, raw versus filesystem 
disk usage, and synchronous I/O. 

Session: Distributed 
Computing 

Summary by Carolyn M. Hennings 

Design and Implementation of an 
Administration System for Distributed 
Web Server 

C. S. Yang and M. Y. Luo, National 
Sun Yat-Sen University 

This presentation, winner of the Best 
Student Paper award, described a system 
for managing distributed Web servers. 
The components of the system include a 
control interface, a controller, broker, 
agent, and remote console GUI. The con¬ 
trol interface is used by system adminis¬ 
trators for turning distribution on and 
off, adding and removing nodes, manag¬ 
ing Web content, and reading statistical 
information. The controller is a Java 
application that runs in the background 
on the distributor and responds to sys¬ 
tem-administration requests. The broker 
is a standalone Java app on each server 
that consists of an agent and monitoring 
thread. The agent performs the delegated 
task on requested nodes such as adding 
and deleting files, searching for a file, and 
analyzing log files. The remote console is 
used for executing management opera¬ 
tions and interacting with the controller. 
The presenters described the system as 
extensible and applicable to any Web site. 


Session: Networking 

Summary by John Talbot 

MRTG - The Multi Router Traffic 
Grapher 

Tobias Oetiker, Swiss Federal Institute 
of Technology 

MRTG is one of the more valuable free 
tools in use today in thousands (a very 
conservative guesstimate) of network and 
Internet sites around the globe. Tobias 
Oetiker, MRTG’s creator, maintainer, and 
chief cook and bottlewasher, originally 
created MRTG in the summer of 1994 
using his free time on a hobby project to 
analyze network traffic at the Montfort 
University. In the short time since then, 
MRTG has established its place as an 
essential tool for network monitoring. 

Unfortunately Oetiker is ending his sup¬ 
port of MRTG because of personal and 
professional time constraints. He deliv¬ 
ered this news with a touch of sarcasm 
and modesty by stating that he “some¬ 
times wonder [s] if no one can program” 
since it is “just a Perl script.” Oetiker has 
done an excellent job of developing and 
maintaining the MRTG code base, and 
his decision will mean a sad loss for the 
progression of this marvelous tool. 

For the meat and potatoes of the techni¬ 
cal discussion, Oetiker concentrated on 
new improvements to the round-robin 
database (RRD) management of the latest 
release of MRTG (called MRTG-3). RDD 
enhancements include the ability to store 
multiple data sources in parallel and a 
break between the database-storage and 
the graph-generation interface, which is 
now called rrdtool. He noted that these 
improvements alone have greatly 
improved the performance of the MRTG 
data-logging capabilities. 

There was some discussion about using 
other databases, such as Oracle, to man¬ 
age the databasing (RRD) and graphical 
functions (rrdtool) of MRTG for large 
datasets and networks. Oetiker was not 
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sure if this would provide any major per¬ 
formance advances over the current 
method. RRD uses a “lossy database” 
topology in which older data is statistical¬ 
ly averaged over spreading sample rates 
as data ages. Using this method of data 
management, the RRD is able to main¬ 
tain a fixed-sized dataset. 

While the latest performance enhance¬ 
ments are great news about advancement 
in the MRTG package, it is sad news that 
Tobias is limiting his future involvement 
with MRTG. 

Wide Area Network Ecology 

Jon T. Meek, Edwin S. Eichert, and 
Kim Takayama, American Home 
Products Corporation 

This was a good nuts-and-bolts discus¬ 
sion of how to improve your WAN per¬ 
formance. Jon Meek and his team con¬ 
cisely and effectively described practical 
and innovative solutions for analyzing 
and enhancing network performance at 
the American Home Products Corp. 

Monitoring basics, such as tcpdump and 
Perl scripting, were some of the methods 
used to measure WAN performance of 
the round-trip times (RTT), committed 
information rates (CIR), and reliability of 
these parameters across numerous WAN 
direct- and virtual-circuit connections. 
For more unusual problems, Meek and 
his team looked directly at packets and 
protocols on the WAN and performed 
system monitoring of process utilization 
and file sizes to gather more process data. 
Often, network-performance measure¬ 
ments and diagnostics were limited in 
their detail by the complexity of the pri¬ 
vate section of the frame-relay cloud of 
their WAN provider. Some solutions to 
this dilemma were to obtain circuit and 
network maps from their frame-relay 
provider(s) and insist on read-only access 
to the SNMP port of the frame-relay 
routers to be able to quickly map and 
identify stops and bottlenecks in the 
WAN architecture. 


Other core essentials to network perfor¬ 
mance were also checked. Redundant 
successive database queries, large 
PostScript files, email attachments, file 
sharing over the WAN, and SQL network 
applications were major sources of band¬ 
width utilization. Hard disks were imple¬ 
mented on many of the network printers 
to handle redundant headers and footers, 
such as company logos and graphics. This 
was a particularly novel idea since it 
enabled local caching of redundant data 
at the destination point (the printer). It 
was found that Web applications used far 
less bandwidth than the SQL applica¬ 
tions, since the Web applications needed 
to transmit only the interface instead of a 
client/server database link. 

Meek and his team took advantage of a 
packet prioritization determined by pro¬ 
tocol and “weighted fair queueing” to 
lessen the severity of “bandwidth hogs.” 
Although the use of access-control lists 
(ACLs) and special packet handling 
introduces extra load on the network 
routers, it enabled interactive network 
sessions to take precedence in the net¬ 
work bandwidth, thereby giving the 
appearance of better response by online 
applications such as Telnet and Web 
interfaces. 

In one instance, a WAN circuit was 
upgraded from a CIR of 128KBps to 
256KBps and the RTTs markedly 
increased, degrading network perfor¬ 
mance. It was found that the frame-relay 
provider actually routed the new connec¬ 
tion upgrade over a more complex set of 
routes in the WAN just to get the “faster” 
circuit connected to Meek’s site. If Meek’s 
team did not statistically and periodically 
analyze network-performance links, they 
would have had no idea where the trou¬ 
ble was, since their frame-relay provider 
did not do any network performance 
monitoring and analysis, only up/down¬ 
time status. Meek’s analysis data was 
enough to give his frame-relay provider 
the impetus to get working on the solu¬ 
tions. Meek stated that many frame-relay 
providers have this same problem. 


Further analysis of WAN RTTs and RTTs 
of Internet-bound connections showed a 
much lower RTT for Internet connec¬ 
tions than for WAN connections. This 
left a few questions about the possibility 
of using Virtual Private Network (VPN) 
technology routed over the Internet to 
handle some of the currently poor-per¬ 
forming WAN links. 

Automatically Selecting a Close Mirror 
Based on Network Topology 

Giray Pultar _ 

Giray Pultar discussed issues involved in 
automatically redirecting queries to an 
HTTP server on the basis of an HTTP 
client’s proximity to the nearest HTTP 
mirror server. 

Automated mirror selection would be of 
great value to sites that have multiple 
Internet presence points across large geo¬ 
graphical areas. An automated mirror- 
selection service or tool would provide a 
single-presence appearance to the client 
host and greatly reduce the need for a 
user to manually choose a mirror site 
from a lead page or site. 

Giray made suggestions for implement¬ 
ing such a system and noted some defects 
of both client- and server-side implemen¬ 
tations of close-mirror selection. Java 
(software overhead) and traceroute 
(routing difficulties due to propagation 
delays and return-trip connections to the 
client host through an origin firewall) are 
limited as client-side solutions. In addi¬ 
tion, the traceroute method could con¬ 
ceivably add large delays for sites with 
numerous mirroring sites. 

Giray’s approach to solving the close- 
mirror problems is to build a mirror 
table of known networks and relate them 
to the geographically “closest” mirror. 
Such a table, if based on all IP network 
address combinations, would be massive 
and difficult to construct. How can each 
network be correctly identified and cate¬ 
gorized? How can physical “closeness” be 
determined when so many ISPs have 
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multiple redundant links and dynamic 
failover routes? 

By identifying collections of networks as 
autonomous systems, the definition of 
“which mirror to use” becomes less com¬ 
plicated. The Internet routing registry 
(IRR) databases are then used to compile 
a correlation table which a server can use 
to redirect a client to a “closer” site based 
on the client s IP number. Two scripts 
were developed for building the close- 
mirror tables. Script closest.cgi is the 
CGI interface to be called by the server to 
determine the “nearest” location. The 
script mkmirrortable contacts the IRR 
databases and compiles the correlation 
table. 

Some caveats still apply to the develop¬ 
ment of this technology. There were 
questions about “what should be” and 
“what is” when using AS paths, since 
there is no hard relation between an AS 
path and a real physical network route. 
Also, expansion delays depend on the 
geographical span and size of a particular 
AS, which can vary in real physical size 
and distance within the definition of the 
AS itself. Other issues, such as load bal¬ 
ancing and mirror-site downtime, pose 
several challenges to the development of 
this new technology which, once solved, 
will prove to have highly useful applica¬ 
tions not primarily limited to HTTP 
redirection. 

The close-mirror package can be found at 
<ftp://ftp.coubros.com/pub/lisa98>. 

Session: Infrastructure 

Summary by Carolyn M. Hennings 

What to Do When the Lease Expires: A 
Moving Experience 

Lloyd Cha, Chris Motta, Syed Babar, 
and Mukul Agarwal, Advanced Micro 
Devices, Inc.; Jack Ma and Waseem 
Shaikh, Taos Mountain, Inc.; Istvan 
Marko, Volt Services Group 

Chris Motta related his experience in 
moving approximately 1,000 machines 


and 220 users from a single building into 
two different buildings. He listed some of 
the things that helped the move go 
smoothly. This list included a well- 
defined scope, using email to communi¬ 
cate during the planning phase, organiza¬ 
tion and planning, a central command 
center, allowing extra time for unforeseen 
problems, blanket purchase orders with 
key vendors, new networks staged and 
tested in advance, and insisting that man¬ 
agers and users were not present during 
the move. Some things that hindered the 
operations were insufficient checking of 
scripts and lists, inaccurate audit from a 
vendor, poor estimate and execution 
from movers, poor estimate/execution 
from a fileserver vendor, lack of working 
phone lines, weekly bureaucratic meet¬ 
ings too far in advance of the move, and 
not getting enough sleep during the 
move itself. Motta made the following 
suggestions: Have independent verifica¬ 
tion of scripts and audit; have a single 
person in charge of controlling the entire 
move and making key decisions; have 
laptops with network cards available for 
use as terminal emulators; have cellular 
phones and/or radios available for every¬ 
one, and plenty of spare cables and 
adapters. 

Anatomy of an Athena Workstation 

Thomas Bushnell, BSG; Karl Ramm, 
MIT Information Systems 

Thomas Bushnell described an academic¬ 
computing environment with approxi¬ 
mately 30,000 users and 1,000 worksta¬ 
tions. The workstations are located in 
public clusters - libraries and hallways - 
as well as in faculty and staff offices and 
dorm rooms. They are all standard UNIX 
workstations configured for a single user 
and serial reuse. Bushnell described the 
concept of “lockers” - storage areas speci¬ 
fied for a particular use such as a home 
directory, packages of software, or com¬ 
mon areas for collaborative efforts by 
groups of people. The “lockers” support 
the release cycle for operating systems 
and software updates. A group of “system 


packs” made up of “lockers” comprise the 
operating system and other software lay¬ 
ered on top of the OS. Machines are 
identified as parts of clusters that deter¬ 
mine at what point in the release cycle 
new software will be loaded. An “auto¬ 
update” facility allows for these lockers of 
software to be loaded as the machines are 
booted. The presenters concluded with 
the following observations: the security 
model gives clarity; the serial reuse model 
presents problems with time sharing and 
long-running batch jobs; and the hands- 
off auto-update and installation allows a 
team of 10 system administrators to sup¬ 
port the 30,000 users and 1,000 worksta¬ 
tions. 

Bootstrapping an Infrastructure 

Steve Traugott, Sterling Software and 
NASA Ames Research Center; Joel 
Huddleston, Level 3 Communications 

Steve Traugott provided an insightful 
overview of the steps for creating and 
managing a solid infrastructure. Some 
key steps are determining how version 
control is going to be done and setting up 
a “gold server ” With this foundation, 
other infrastructure elements such as 
installation tools, directory and authenti¬ 
cation services, network fileservers, client 
file access, and configuration manage¬ 
ment can be implemented. These steps 
and others are detailed in the paper along 
with a graphic describing the order in 
which they should be performed. This 
architecture has advantages in disaster 
recovery, software distribution, and low¬ 
ering total cost of ownership. Traugott 
concluded the presentation with the 
observation that when it comes to defin¬ 
ing an infrastructure, the role being filled 
is larger than “system administrator” and 
might more accurately be called “system 
architect.” 


8 


Vol. 24, No. 1 {login: 



Session: Distributing Software 
Packages 

Summary by Chastity D. Arthur 

mkpkg: A Software Packaging Tool 

Carl Staelin, Hewlett-Packard 
Laboratories 

Carl Staelin has developed a remarkable 
tool to allow software publishers to easily 
create installation packages. Staelin 
pointed out that the industry has focused 
on the end users and systems administra¬ 
tors, allowing them to easily install and 
uninstall software, and has not focused 
on the first step, the software distributor 
who has to create the binary installation 
package. With Staelin’s tool, mkpkg, the 
software distributor can add a descrip¬ 
tion of the package, develop manifests, 
include certain dependencies, create 
install and uninstall scripts, and cus¬ 
tomize the post-installation, mkpkg can 
take as little as three minutes to complete, 
provided the software is ready for distrib¬ 
ution. 

Staelin addressed the portability of 
mkpkg. It was developed on HP-UX and 
uses HP-UX-specific commands. He has 
successfully ported mkpkg to ninstall, 
update, and SD-UX. His next conquest 
will be RPM, but his work has slowed for 
lack of time, mkpkg is available at 
<http://www.hpLhp.com/personal/CarLStaelin/ 
mkpkgx 

SEPP - Software Installation and 
Sharing System 

Tobias Oetiker, Swiss Federal Institute 
of Technology 

Tobias Oetiker and his IT support group 
(ISG) saw the need to provide a software- 
installation tool that would reduce the 
repetitive task of installing software and 
configuring the systems throughout the 
various departments in the institute. 

They were also looking to develop a tool 
that would retain some independence in 
the installations. The ISG tested software- 


distribution tools already on the market, 
comparing Red Hat’s Package Manager, 
GNU Stow, Depot-Lite, and LUDE, to 
name a few. None of these tools met their 
requirements, nor did they use wrapper 
scripts - but the ISG did discover that in 
a mix of all these tools and a few of their 
own ideas lay exactly the features they 
needed. Thus SEPP came into existence. 
SEPP provides both a clean system for 
system managers to use and a user- 
friendly environment. It is currently sup¬ 
ported only on Solaris and Irix. 

SEPP includes a number of system-man¬ 
agement features. The subdirectory tree 
provides clean encapsulation to all files of 
the same distribution; a special directory 
(called SEPP) in each software subdirec¬ 
tory houses a description of the contents 
along with the startup wrapper script, 
start.pl; the automounter tool, using 
/usr/pack, helps to ensure paths during 
compiles; the packages’ binaries are actu¬ 
ally symbolic links to /usr/sepp/bin, 
which points to stub scripts; Perl scripts 
start up the wrapper script; and a unique 
name field is generated for each software- 
package distribution. Oetiker was not 
only very proud of SEPP’s system-man¬ 
agement features, he also highlighted the 
user features. One of SEPP’s most conve¬ 
nient features is that the user only needs 
to add /usr/sepp/bin to the PATH vari¬ 
able. The ISG also developed both Web- 
based and manual-page-based SEPP doc¬ 
umentation. SEPP also allows for multi¬ 
ple versions of the same software distrib¬ 
ution using suffixes appended to the exe¬ 
cutable names. 

With SEPP’s reliance on the auto¬ 
mounter, user applications that are 
required during bootup will cause prob¬ 
lems. The ISG is currently addressing that 
issue by adding a feature that enforces the 
bootup applications to be mirrored to the 
local machine. 

SEPP is distributed under the GNU 
General Public License and can be 
obtained from <http://www.ee.ethz.ch/sepp>, 


where there is also information on the 
SEPP mailing list. 


Synctree for Single Point Installation, 
Upgrades, and OS Patches 

John Lockard, University of Michigan; 
Jason Larke, ANS Communications 


Synctree is a system-administration tool 
developed for a large network requiring 
frequent OS or software updates and 
security patches. Lockard and Larke had 
two goals in mind - system security and 
uptime. Synctree is capable of holding 
the network’s complete configuration in a 
secure, readable format. The idea behind 
Synctree is to bring a machine up on the 
network and “sync” it to the templates for 
the architecture so defined. 


Comparing Synctree to cloning, the 
authors stated that although a clone 
could be made that meets your require¬ 
ments, each time an update is added to 
that architecture a new clone would have 
to be established. Another comparison 
was made to rdist in that rdist relies on 
each machine being up and connected to 
the network when you run your update. 
Under the direction of Paul Howell, the 
University of Michigan’s Computer Aided 
Engineering Network group wanted to 
create a utility that provides verification 
of widely distributed patch installations 
and ensures that files prone to hackers 
are in their expected state. 

Synctree’s template permissions are based 
at the client level, and only the root user 
of that client can call a sync and order 
the classes the client syncs to. Synctree 
relies on a server, and any other work is 
copied in downloads to the clients. 
Synctree also allows images to overlay 
each other, like GNU’s cfengine. With 
this feature, the client actually builds the 
final picture before implementing any 
changes. Synctree has only one configura¬ 
tion file, /etc/hostconfig, where 
each class of machines is listed. Synctree 
goes down to the level of file-to-file 
comparison. 
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In closing their presentation, Lockard 
and Larke talked of future features they 
would like to incorporate into Synctree. 
One update for the near future is allow¬ 
ing Synctree to install software packages 
normally found on the network to the 
local hard drive. Currently Synctree relies 
on AFS, which not everyone has or 
wants; a future goal is to adapt Synctree 
to another secure copying system, such as 
krcp. 

A Synctree sample is available for non¬ 
commercial use only at 
<ftp.math.lsa.umich.edu/pub/Synctree/>. 

Session: Mailing Lists 

Summary by Brian Kirouac 

Mailman: The GNU Mailing List 
Manager 

John Viega, Reliable Software 
Technologies; Barry Warsaw and Ken 
Manheimer, Corporation for National 
Research Initiatives 

Have you ever subscribed to a list and 
later realized that you forgot what type of 
mailing list it was and how to unsub¬ 
scribe? As a list owner it would be nice to 
add a footer to each message that 
describes the process for unsubscribing. 
Viega, Warsaw, and Manheimer wanted 
to add just such a footer to a majordomo 
mailing list. This worked fine for individ¬ 
ual messages. The problem was each mes¬ 
sage with its footer was put in the digest, 
so there were multiple copies of the foot¬ 
er in the digest. 

They started looking at different mailing- 
list-management software packages for 
something that would allow the user to 
subscribe or unsubscribe quickly and eas¬ 
ily, and would allow the list owner to 
manage the list. MajorCool was consid¬ 
ered “cool” but limited. 

Mailman came of this. Mailman offers a 
Web-based user interface that allows list 
management on three levels: user, list, 


and site. It includes email-based com¬ 
mands, but the Web based interface is the 
driving force. A user or owner can sub¬ 
scribe or unsubscribe from a list as well 
as choose between live and digest modes. 
A list owner can edit the list’s Web page 
and set various list options. 

Drinking from the Fire(walls) Hose: 
Another Approach to Very Large Mailing 
Lists 

Strata Rose Chalup, Christine Hogan, 
Greg Kulosa, Bryan McDonald, Bryan 
Stansell, Global Networking and 
Computing, Inc. 

Strata Rose Chalup presented the 
authors’ experience moving the 
“Firewalls” mailing list. The original serv¬ 
er used large ISPs to do mail relaying 
instead of doing the delivery itself. When 
GNAC took over the list, it did not have 
the same relationships with ISPs, and 
thus the new server had to deliver the 
mail. 

The typical two-queue system did not 
function well enough. The outbound 
queue was growing faster than mail was 
getting delivered. The problem was that 
majordomo was creating a single send- 
mail queue file generated with 4000+ 
addresses in the RCPT line. 

They created a Perl program run every 
five minutes out of cron, called qsplit. 
This takes the original queue file and 
splits it up into easier-to-spool chunks. 
Each chunk having a specified number of 
recipients, they chose 25. To keep the 
uniqueness of queue file names, each 
chunk has a sequence number appended 
to the original name. These are then 
spread through 10 different queue direc¬ 
tories. 

Each queue directory runs a separate 
instance of sendmail to process the 
queue. A process called spawn is respon¬ 
sible for keeping these sendmail processes 
running. Spawn is smart enough to keep 


the system busy but not have it swapping. 
This way as much mail is delivered as fast 
as possible. 

Request v3: A Modular, Extensible Task 
Tracking Tool 

Joe Rhett, Navigist 

This was a presentation on some of the 
modifications and extensions that have 
been made to Request. 

Some of the problems of not having a 
good tracking tool are: task history is 
usually stored in human RAM, thus 
prone to loss; handoffs are not always 
handled well; there’s little or no informa¬ 
tion to justify staff. Requirements of a 
good task tracking tool are: track entire 
history of task; do not slow down admins 
who are using the tool; support almost 
any operating system or platform; work 
well from remote; be easy for untrained 
users to access; acquire statistics. 

Commercial applications may fit some of 
these requirements, but they are expen¬ 
sive, require a lot of training, and don’t 
usually support all platforms. Free appli¬ 
cations are generally not updated often, 
require UNIX-like skills, and don’t always 
have Web and email interfaces. 

The previous versions of Request had 
several problems. First, they were not 
year-2000 compliant. Parts of the code 
aggregated Perl 5, and small changes 
required many fixes. Most problems 
related to dispersion. The design goals of 
the new request were to fix these prob¬ 
lems and to allow others to add code eas¬ 
ily. It actually resulted in fewer lines of 
code. 
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INVITED TALKS TRACK 

Zero to LISA in One Year 

Brent Chapman, Covad 
Communications Company 

Summary by Chastity D. Arthur 

Brent Chapman explained the successful 
and unsuccessful decisions made as the 
Silicon Valley startup, Covad, coped with 
its one-year growth from one region with 
50 people to six regions with 400 people. 
He was a member of the IT department, 
faced with continually scaling and sup¬ 
porting the network and responding to 
systems demands. He discussed the ongo¬ 
ing process of planning every detail pos¬ 
sible and attempting a proactive 
approach to situations. “No plan survives 
reality, but it’s a start,” he said. 

A startup company must recognize the 
challenges, both obvious and hidden. The 
obvious issues are: keeping up with the 
growth; getting ahead of the growth; 
attracting and retaining top talent; devel¬ 
oping adequate and scalable systems; 
maintaining daily development support; 
introducing new tools, services, and con¬ 
cepts to the users; and developing a 
strong infrastructure not just for IT but 
for the entire company. Some hidden 
challenges are: the linear rise in number 
of hours required; growth of users’ 
expectations; loss of volunteers in the IT 
department; and old users being more 
self-sufficient than newer users. In 
Covad’s case, company culture was also a 
factor. When Covad was smaller, commu¬ 
nication was always at peak performance; 
as the company grew, disseminating 
information became more of a challenge, 
and it became harder for older employees 
to find time to work with new ones. 

Chapman discussed the considerations 
that went into site selection (Silicon 
Valley offered many advantages) and 
described the headaches and heartaches 
of surveying site after site. He explained 
how IT planned for the move once a 
location was chosen. What better way 


than to set up a MOCR, a NASA-like 
Missions Operation Control Room. 
Chapman described in depth how the 
success of the MOCR enhanced the suc¬ 
cess of Covad’s move. The first decision 
was to hire trusted contractors to aug¬ 
ment the staff. Chapman then armed 
everyone with a radio and appointed 
flight directors in a rotation that meant 
the MOCR always had a manager. The 
flight plan was simple - make all critical 
decisions in advance. A successful move 
was completed in one weekend, and there 
was still time to leave welcome packages 
in the cubicles. IT left a welcome note for 
the employee, a map of the new building, 
including restrooms and printer loca¬ 
tions, plus a little treat. The MOCR 
remained open as a help desk through 
the first official busy day. Chapman 
described the MOCR as a “great central¬ 
ized success.” 

Chapman then discussed the one 
demand that kept arising - users wanting 
more bandwidth. Although bandwidth is 
the first to be blamed, it is often not the 
problem. IT should help the executives 
understand the cost of more bandwidth, 
help the users understand how to use 
software or choose their software, and 
explain the differences between latency 
and bandwidth. 

Chapman ended his talk with why he 
chose to join Covad as a startup compa¬ 
ny. He wanted to be a part of something 
that could be successful and to have the 
opportunity to work with outstanding 
and experienced people, interesting tech¬ 
nology, and vendors. He closed with the 
observation that maximum productivity 
is nowhere equal to the maximum num¬ 
ber of hours worked. 


Got LDAP? Deploying and Using the 
Lightweight Directory Access Protocol 

Leif Hedstrom, Netscape 
Communications Corporation 

Summary by Brian Kirouac 

Is your directory information starting to 
become overwhelming? Someone sug¬ 
gests LDAP, the Lightweight Directory 
Access Protocol, so you start looking at 
the documentation. One of the first 
things to hit you is that deploying and 
managing a directory server is a compli¬ 
cated task. 

Leif Hedstrom of Netscape gave a good 
talk dealing with some of the issues and 
pitfalls associated with installing a new 
LDAP-based system, based on some of 
the issues Netscape faced when installing 
its LDAP servers. 

Before designing anything, you need to 
establish your goals. The first is easy dis¬ 
semination of information. Two other 
concerns should be the scalability and 
performance of your server. Scalability 
and performance have a direct impact on 
how to design your database tree struc¬ 
ture. 

During the design and implementation, 
you need a manager who will back you, 
and you need to make sure you have all 
departments involved. It was amazing 
how much input the legal department 
had in Netscape’s implementation. 

Several legal concerns can influence what 
data you might be able to include. 
Pictures, home phone numbers, and car 
license plates, for example, might be 
problematic. 

Useful guidelines for planning and imple¬ 
menting LDAP: spend time planning, 
analyzing, and testing the design; select a 
directory-information tree that is as sim¬ 
ple as possible; elect the proper software 
based on your needs. 
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Succumbing to the Dark Side of the 
Force: The Internet as Seen from an 
Adult Website 

Dan Klein, Cybertainment, Inc. 

Summary by Bruce Alan Wynn 

Dan Klein gave effectively the same talk 
as he had at USENIX ’98 in New Orleans, 
without displaying any defensiveness 
about the fact that he is the technical per¬ 
son for a dozen pornography Web sites. 
He went over some of the technical issues 
for maintaining such a site, and noted 
that porn sites tend to have better securi¬ 
ty and adult-verification than some 
banks. The talk was very well attended. 
(No, he didn’t show pictures; the talk was 
PG-13.) 

On the technical side, Klein talked about 
techniques to reduce the load on a Web 
server: load sharing, load shedding, and 
load boosting. Load sharing is basically 
using DNS entries in a round-robin fash¬ 
ion to distribute the load. The main issue 
with this is making sure that all of the 
servers have the same data. Load shed¬ 
ding requires a front-end server that 
hands off initial requests to back-end 
servers that have the real content. The 
problem here, again, is keeping every¬ 
thing in sync. Load boosting is per¬ 
formed on the client side. A lot of sites 
make their money based on the number 
of hits a given URL receives. Thus sites 
will have banners and Javascript pro¬ 
grams that pop up other windows that 
access the same URL. Load boosting con¬ 
sists of turning off Javascript on the 
client to prevent these other windows 
from appearing, thus reducing the 
amount of time it takes to load a page. 

A good practice he mentioned is keeping 
logs. Logs help plan for the future, and 
they help determine possible security 
breaches. And, in the case of legal action, 
they can help cover you if someone false¬ 
ly accuses you of something. 


Branchstart - A Generic, Multi-OS 
Installation Server 

Rory Toma, WebTV Networks, Inc. 

Summary by Chastity D. Arthur 

Rory Toma described his successful 
implementation of a single-architecture 
yet multi-OS network installation server 
on Intel-based platforms. His project isn’t 
actually named Branchstart; he is playing 
on the name of Sun’s product, Jumpstart. 
Toma calls his implementation MOSIP, 
an image- and package-based OS 
installer, successfully tested on Red Hat 
Linux 5.2, NT 4 Workstation, NT 4 
Terminal Server, and Windows 95. 

Toma’s project goals included: minimal 
user interaction, 100 percent predictabili¬ 
ty, easy scalability, and functionality at a 
junior level. He commented that MOSIP 
is reproducible, flexible, and fast to 
install. On the more technical side, 
MOSIP has a binary failure mode; opera¬ 
tors can use the same install server for 
multiple OSes or platforms; and a serial 
console or GUI is optional. Not so invit¬ 
ing is the amount of front-end work and 
the level of knowledge and experience 
needed to set up advanced installations. 

Toma described how he made MOSIP 
come together. For each OS base needed, 
a template machine must be installed to 
acquire the OS image with dd and to 
record software- or hardware-specific 
parameters. He then described what he 
terms “laying down the bits” - basically, 
booting Linux with NFS root filesystem 
and having an installation script run 
automatically. Toma chose to replace init 
with his own script. To finish, he 
described “modifying the bits.” This is the 
point at which the administrator would 
modify the IP address, create auto-login 
scripts, and install LILO. 

In closing, Toma discussed his next proj¬ 
ect, Internet OS Installation Server 
Project (IOSISP), which will take MOSIP 
to the next level: installation of free OSes 
from the Internet. He plans to automate 
the installation of MacOS and Windows 


2000 to include Active Directory and 
Exchange. He would like his project to 
function on nonlntel hardware, specifi¬ 
cally for NT and Linux. His greatest chal¬ 
lenge, he said, is creating a generic NT 
image that would allow modification to 
suit a wider range of hardware. 

To learn more about MOSIP or IOSISP, 
visit <http://www.munitions.com/rory/MOSIP/ 
welcome.html>. 

THE GREAT 

CERTIFICATION DEBATE 

Moderator: Rob Kolstad. Panelists: 

Phil Scarr, GNAC; Scottie Swenson, 
University of Washington; Linda True, 
TRW Space and Electronics-, Bruce 
Alan Wynn, Collective Technologies 

Summary by Carolyn M. Hennings 

If there is a hot topic in the SAGE com¬ 
munity, it is definitely certification. The 
issue has been in the air for a number of 
years, and the SAGE Executive Com¬ 
mittee decided it was time to take action 
toward making a decision to either pur¬ 
sue or drop it. A SAGE Certification 
Subcommittee was formed, and subse¬ 
quently a Certification Advisory Council 
was created. 

The purpose of “The Great Certification 
Debate” was to have a serious discussion 
about the certification issue. Rob Kolstad 
asked the panelists to introduce them¬ 
selves and speak to their positions on the 
issue. On the pro side of the discussion 
were Bruce Alan Wynn and Lynda True. 
The cons were Phil Scarr and Scottie 
Swenson. 

Bruce Alan Wynn expressed his opinion 
in light of the SAGE charter of advancing 
systems administration as a profession. 
The certification project will help with 
the definition of our system-management 
standards and then define the require¬ 
ments for certification. Bruce reminded 
the audience that SAGE uses the term 
“guild” in its name, referring to a struc¬ 
ture in which more experienced people 
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help the less experienced. The certifica¬ 
tion process would provide guidance and 
direction to individuals who don’t know 
where to start. He concluded by saying 
that there are a number of different ways 
to do certification, that some are better 
than others, and that SAGE needs to do it 
right. 

Phil Scarr reiterated concerns expressed 
in his ;login: article (“When Worlds 
Collide,” August 1997). He suggested that 
the best administrators come from uni¬ 
versity programs where there is an educa¬ 
tion focus rather than a certification 
focus. Experience is a better indicator of 
ability than certification. Certification is 
touted as a way to hire but is not effec¬ 
tive. 

Lynda True explained that her organiza¬ 
tion has been certifying UNIX adminis¬ 
trators for approximately two years. 
Management recognized inadequate sys¬ 
tem administration support and lack of 
training to be potential threats to infor¬ 
mation security and Internet availability. 
Although the process has been painful, 
some benefits have been that hiring has 
become easier and salaries have risen. She 
suggested that the peer-review portion of 
the certification process was an impor¬ 
tant aspect. 

Scottie Swenson commented that certifi¬ 
cation might be good if done correctly. 

He commented that most vendor-spon¬ 
sored certification programs have little 
value. He expressed concerns regarding 
the difficulty of managing a certification 
program. A focus on education should be 
a priority over certification. 

Questions and comments from the audi¬ 
ence expressed concerns regarding how 
the certification process might work. 
Numerous people brought up the issue of 
education and how it relates to certifica¬ 
tion. Suggestions included looking at the 
certification processes used by the med¬ 
ical, project-management, and aviation 
professions. An important point brought 
up by one participant was that if SAGE 


doesn’t certify system-management pro¬ 
fessionals, some other organization will. 

If that happens, we run the risk of having 
to live with something that isn’t the best 
and doesn’t quite work. 

INVITED TALKS TRACK 

Security as Infrastructure 

Tom Perrine, San Diego 
Supercomputer Center 

Summary by Kurt Dillard 

Tom Perrine convincingly asserted that 
an effective system administrator must 
address security at all seven network lay¬ 
ers as well as two others, the economic 
and political layers of your organization. 
He summarizes effective approaches as 
“building fences” while everyone else 
stays busy “shooting rabbits.” A long-term 
solution to effective security will take 
time to implement, and a few “rabbits” 
may get in while you build it, but the 
final result will be much more satisfacto¬ 
ry and manageable. Perrine suggests that 
you undertake implementing a secure 
infrastructure by first defining goals - 
figure out what needs to be secure and 
rank those items by importance. Then 
decide how you will respond to different 
types of attacks, automate dealing with 
unsophisticated attempts, and don’t even 
waste time investigating them personally. 
Automate dealing with other types of 
probes and analyze the data personally. 
Figure out what attacks are so sophisti¬ 
cated that you will have to deal with 
them personally from start to finish. 
Basically, this is risk analysis - figuring 
out what is important, what the threats 
are, and how much time you need to 
spend protecting what’s important from 
the different types of hosts. By the way, 
automate or die, because if you cannot 
scale your solution it will fail as your net¬ 
work grows. 

Perrine then reviewed current attack 
methods and ways to deal with them. 
What’s being used? Every approach you 


have ever heard of. Even the weaknesses 
with known fixes are effective for hacking 
because most sites never bother to apply 
the correct fixes properly. The biggest 
threats are automated exploits that allow 
attackers to probe numerous hosts very 
quickly, and any type of authentication 
that requires a password transmitted as 
plain text. 

To deal with these attacks you need to 
automate as much as possible in your 
environment. Implement CfEngine or 
SMS to allow for quick host installs and 
simplified patches and upgrades. Set a 
security policy and automate a method of 
auditing all hosts to ensure that they are 
meeting your policy. Segregate the weak 
hosts from the rest of the network by 
placing them behind filtering routers. 
Automate the detection of anomalies and 
as much as possible automate resolutions 
for those anomalies. Don’t forget to edu¬ 
cate everyone who uses your network; 
“social engineering” is a very effective 
method for hacking sites. Finally, remem¬ 
ber that you have to win the support of 
your users and your management chain. 
Build your secure infrastructure slowly so 
that your users have a chance to get used 
to it and so that you have time to verify 
that everything is working at each step. 

PRACTICUM TRACK 

Teaching Systems Administration 

Chair: Lee Damon, QUALCOMM, Inc. 
Panelists: David Kensiski, Digital 
Island, Inc.; David Kuncicky, Florida 
State University; Daniel Klein, 

USENIX Association; Matt Shibla, 
Montgomery Blair High School 

Summary by Chastity D. Arthur 

Consensus at this practicum was that no 
one has all the answers to teaching sys¬ 
tem administration, and one particular 
method is not going to solve this issue. 
Only a combination of school programs, 
extension programs, in-house training, 
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on-the-job training, and vendor courses 
is the answer for today. 

David Kensiski appeared to be in favor of 
a combination of vendor courses and in- 
house training. His answers were clear 
and concise. One audience member asked 
him what he did with a junior employee 
who just wasn’t grasping the concepts. He 
politely stated that there was really only 
one option - either find them something 


they can do in another area or terminate 
them. (This really only brings up another 
question, what if it’s the teaching 
method?) 

David Kuncicky thinks universities need a 
way of defining the appropriate levels of 
system administration in order to 
advance a student, and he compared the 
teaching methods and coursework with 
those for programming languages. 
Kuncicky presented his department’s cur¬ 
rent course design. FSU’s program is a 
mix of books, courses taught on campus 
by FSU instructors, and third-party train¬ 
ers. He defined the program by step levels 
in Systems Administration Proper, SA 
Tools, SA Networking, and SA 
Applications. Kuncicky also mentioned 
that what may separate the different uni¬ 
versities is state legislation that mandates 
the number of hours, which led FSU to a 
system-administration master’s track. He 
was not necessarily an advocate of the 
master’s track, but FSU’s goal was simply 
to get a program started. 

14 


Daniel Klein, tutorial coordinator for 
USENIX, brought up a major concern of 
the system-administration industry 
today: How do you know when or which 
training is appropriate? He discussed the 
assumptions that must be made prior to 
proceeding with any type of training. 
Klein mentioned that a certain amount of 
prior experience here and there is needed 
and that the potential trainee must be 
self-driven, willing to read, use references, 


and know when to ask for assistance. 
Klein was an excellent communicator and 
he immediately grabbed the audience 
when he said, “Training does not fix your 
problems.” He followed that up with a 
concise statement that system adminis¬ 
trators can be taught with the see-one, 
do-one, teach-one theory. He pointed out 
that indeed vendor training feeds the cur¬ 
riculum, but it is also a general assump¬ 
tion that the instructor knows the course 
materials but may not necessarily be an 
expert on the subject. This is what makes 
USENIX’s curriculum flexible; the 
instructors are the gurus and as technolo¬ 
gy changes, advances, or grows, so will 
the course. 

The last panelist, Matt Shibla, was there 
to discuss the Maryland Virtual High 
School established through Montgomery 
High School. This is an online curricu¬ 
lum downloaded from Cisco Network 
Academy to the high school’s private net¬ 
work. Shibla stated he felt the program 
was 60 percent generic and 40 percent 



vendor-related. The main issue that arose 
from the audience was that this type of 
course is limited and teacher-deficient. 
Shibla pretty much agreed and did not 
offer any insight into developing the pro¬ 
gram further other than trying to get 
teachers certified. 

This practicum was not set up to provide 
the answers but to give the panelists and 
audience ideas on different directions 
they could take in designing their own 
training methods. The audience was pri¬ 
marily focused on the motivation of 
trainees and teachers and the manage¬ 
ment issue of “you train them, they 
leave.” 

Network Administration and Remote 
Computing 

Moderator: Rob Kolstad. Panelists: 
Pete Lynch, Jyra Research; Shaula 
Yemini; Oljad Singh, System 
Management Arts 

Summary by Kurt Dillard 

All three speakers presented information 
about the network-monitoring tools that 
their companies have developed. Oljad 
Singh’s approach is to focus on critical 
services and warn of impending and 
existing problems. His company has 
installed a server, running their monitor¬ 
ing software, that executes ghost transac¬ 
tions every few minutes on each server 
being monitored. The monitoring soft¬ 
ware times each transaction and over 
time learns what the “normal” perfor¬ 
mance signature is of each application on 
each server at different times of the day 
and different days of the week. When sev¬ 
eral transactions in a short period of time 
fall too far outside of the “normal” signa¬ 
ture, the monitoring software automati¬ 
cally notifies an operator of a possible 
problem. 

Pete Lynch promoted a product from Jyra 
that proactively monitors defined busi¬ 
ness and performance goals. He asserted 
that service-level agreements don’t ade¬ 
quately measure the user’s experience, 
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while Jyra’s monitoring solution does so 
by measuring the response time of appli¬ 
cations using a scheduled polling agent to 
see how each application server is per¬ 
forming. Their scalable solution utilizes 
distributed Java agents and provides 
automatic reporting, summaries, and 
exception alarms. 

Shaula Yemini has another proactive real¬ 
time management tool, SMARTS, which 
diagnoses problems before they have had 
an impact on the network, allowing them 
to be resolved before users realize that 
anything has gone awry. SMARTS creates 
a “codebook” that combines generic 
models for each network object with the 
customers specific topology to create 
“problem signatures.” When the problem 
signatures are detected the system raises 
an alert. Her firm already has models for 
a wide variety of network hardware, and 
the codebook can be created by connect¬ 
ing their system to some common net¬ 
work-management systems such as 
Open View and NetView. The codebook is 
automatically updated every time the 
models or topology change. 

WORKS-IN-PROGRESS (WIPS) 

Coordinator: Peg Schafer, Harvard 
University 

Summary by John Talbot 

The WIPs Practicum was more than the 
advertised “pithy” display of current tech¬ 
niques and issues. It was a nonstop blitz, 
in concise 10-minute samples, of interest¬ 
ing and thought-provoking descriptions 
of “real-world” problems and solutions. 
The WIPs were pitted against one anoth¬ 
er with a real whip bestowed upon the 
winner, who was determined by group 
applause. 

Steven Nelson’s “Multiplatform Storage 
Area Network” WIP was a nuts-and-bolts 
discussion of managing a network of over 
1.6 TB of database and data sources in a 
24x7 environment. Nike s original fiber- 
channel network was initially spread out 
over all storage areas, but problems per¬ 


sisted because of shared data paths for 
both the backup and data-delivery ser¬ 
vices. They have used the EMC array and 
parallel pipes to have transparent access 
to data dumps without interfering with 
network and system resources. They still 
have large-size backups and multiple 
filesystems for data storage. They are 
looking into ways of using the Veritas 
volume manager filesystems on their 
EMC array to have a commonly mount- 
able local vxvm filesystem between het¬ 
erogeneous platforms. 

John Buckman <john@lyris.com> presented 
a unique implementation of email as a 
mission-critical application. Instead of a 
centralized MTA and MDA, Lyris opted 
for a thin multithreaded mail daemon 
that uses SQL as the message store. This 
enables his site to utilize the text search¬ 
ing and user access of the existing SQL 
structures to manage email. Also, they are 
able to deliver an email interface using 
HTML and Tel more easily with the 
existing SQL structures. 

Lowell Snyder <lsnyder@ptc.com> present¬ 
ed another good email implementation. 

(I might be a little biased since I was a 
cohort of this WIP, but it did come in a 
close second.) Lowell presented some of 
the work done at Parametric Technology 
to remove the standard UNIX aliasing 
from the central MTA and institute 
LDAP hooks into the internal mail 
exchangers to directly deliver mail based 
on the company’s managed LDAP data¬ 
base. A motive was that a majority of 
users were consolidated on a central 
POP/IMAP MDA, and the management 
of simple UNIX mail aliases became 
cumbersome as the user base grew to 
over 4,000. Snyder described the code 
changes required on the sendmail and 
LDAP daemons necessary to invoke the 
features and handle several exceptions for 
whitespace and parsing symbols not gen¬ 
erally understood or interpreted by gen¬ 
eral sendmail parsing rules. 

Tom Limoncelli <tal@lucent.com> presented 
“Tricks you can do when your firewall 


is a bridge.” This WIP stemmed from a 
project Limoncelli had at Bell Labs/Lucent 
when cutting over backbone routers to 
new firewalls to the Internet. The prob¬ 
lem was originally complicated by the 
fact that he didn’t have access to all the 
routers in the schema and still needed to 
implement the firewall changes transpar¬ 
ently. Of note was the fact that when the 
firewall acts as a bridge and doesn’t have 
an IP number, it is less likely to be subject 
to hacking attempts since it has no 
TCP/IP-bound interface. See 
<http://www.bell-labs.com/usr/tal>. 


Charles Tatum <tatum@nswc.navy.mil> of the 
U.S. Navy presented “Computationally 
Expensive Intrusion Prevention.” He 
focused mainly on his modifications to 
the popular Crack program. He surmised 
that a massive amount of time was being 
used by the Crack code to test less proba¬ 
ble circumstances of password guessing. 
His new approach was to apply common 
rules first. For example, he found that 
most people don’t use more than one 
word for their password, so applying a 
number of permutation rules to the pass¬ 
word guessing before trying “obvious” 
guesses makes running Crack computa¬ 
tionally expensive. Also, implementing a 
separate dictionary of names ahead of the 
standard dictionary will weed out more 
obvious guesses. Reducing the salt guess¬ 
es also lowers process runtime. Tatum’s 
approach manipulated the original Crack 
encrypt and compare looping structure 
from Diet—>Rules—>Users—»Salts to 
Rules—>Dict—»Users—>Salts and imple¬ 
mented his common-guess strategies to 
greatly lower computing times. 

Geoff Halprin 

<geoff.halprin@sysadmin.com.au> of SysAdmin 
Group, winner (by a hair over Lowell 
Snyder) of the WIP prize, gave a stunning 
talk about the “Taxonomy of Best 
Practice.” This was one of the clearest 
talks classifying and describing what 
sysadmins do that I have heard in a long 
while. Halprin not only explored the 
need for sysadmins to understand their 
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Geoff Halprin 

own personal technical capabilities, but 
also explained that the user community 
needs to understand what sysadmins do 
and of what they are capable so that- 
sysadmins can be properly matched to 
the environments they support. He 
defined system-administration core com¬ 
petencies as control, organization, protec¬ 
tion, optimization, and planning, and he 
included a diagrammatic breakdown of 
each of these characteristics. He estab¬ 
lished models for organizing these 
responsibilities on a set of five levels, with 
three to five of those levels being consis¬ 
tent across all platforms. The five levels 
are determined by a Capability Maturity 
Model by which certification is based 
upon core competencies. This WIP would 
be a great basis for a future LISA invited 
talk. 

Michael Ewan <michael.ewan@tek.com> of 
Tektronics described using LDAP to cre¬ 
ate printer definitions and determine 
printers by class (e.g., color, resolution, 
paper size) and location. He described 
how the current workstation environ¬ 
ment DISPLAY variable and user profiles 
could be used to reference the “nearest” 
user-default printer. He has also looked 
into ways to use the LDAP information 
base to manage printers. His environ¬ 
ment eased implementation by its homo¬ 
geneous use of Tektronics printers and 
the standardization of UNIX worksta¬ 
tions. 

Andrew Hume <andrew@research.att.com> 
and Tom Scola from AT&T Labs present¬ 
ed “How to Handle Microsoft 


Attachments” in UNIX email. They want¬ 
ed to have a UNIX-based reader that 
would be able to read email attachments 
in realtime. To reduce the need for spe¬ 
cialized windowing software, they devised 
a plan that sends the Microsoft attach¬ 
ment to an NT system that runs a 
PostScript converter and sends it back to 
the Xwindows-based MTA. Unfortuna¬ 
tely, time was running short at this point 
of the WIP session and many of the 
details were abbreviated before Hume 
could complete his WIP. 

BOFS 

Summary by Douglas Stewart 

Variable Length Subnet Masks on 
TCP/IP Networks 

Mike Andrews 

The basics of Class A/B/C and their cor¬ 
responding netmasks, and reserved class¬ 
es for testing and internal use, were cov¬ 
ered. Using all of your addresses in a sin¬ 
gle network is wasteful, especially if your 
company has a Class A address allocated 
to it and has its network broken up into 
geographically separate chunks. By modi¬ 
fying your netmask, you can break up 
your network into smaller, separate pieces 
that can be routed separately. A useful 
example is an ISP that resells large num¬ 
ber of Tls. Typically you’ll have a subnet 
composed of only the router on each 
side. By setting your netmask to 
255.255.255.252, you break your network 
into 64 subnetworks with four addresses, 
which include a network address, the 
broadcast address, and two usable host 
addresses you can use for the routers. 
Routing protocols and commands for 
troubleshooting routing problems were 
suggested. Some light reading (RFCs 950, 
1918, and 1878) was suggested. 

BSDI 

Doug Urner 

Most of this presentation was on new 
features of BSDI 4.0. Filesystem code 
updates: soft update (delays certain file 


operations for speedup, very temporary 
files may never be created); even out 
update; 64-bit file offsets; mount options 
to disable access time updates; mount 
options for sync/asynch writes; and soft 
read-only. Networking improvements: 
PCB lookup hashing; IP address hashing 
for fast virtual hosting; per-address IP 
statistics; and kernel-level packet filter. 
IPv4 enhancements: slow start; conges¬ 
tion avoidance; multi-cast; large win¬ 
dows; MTU discovery; and IPSEC. IPv6 
support! Includes SAMBA, IP/IPX, 

Novell 3.x file and print services, VPN 
support. NFS: v2 and v3 support over 
UDP or TCP; NFS lock daemon. New 
network media: frame relay. SMP: perfor¬ 
mance improvements (user-level process¬ 
es show best improvements); threads are 
all user-level for now but the kernel will 
be threaded in the future. New hardware 
improvements include bootable CD- 
ROMs and plug-and-play Ethernet, 
modem, and sound cards. Binaries are 
now ELF, and the math libraries have 
been proved. There’s a console debugger 
and trace facility called Kdebug and KTR. 
Things to look for in the future: Linux 
binary compatibility, Java application 
environment, finer-grained SMP with 
kernel threads, SPARC port, ATM, and 
channelized T1 and T3 support. 

AFS 

Esther Filderman 

The people who attended this BOF were 
almost entirely AFS users from university 
environments. The first topic was release 
dates for Linux and NT-based AFS 
servers - apparently in February 1999. 
Transarc, which has been bought by IBM, 
was a topic for heavy discussion. It has a 
new CEO, has changed its mind about 
dropping the development of AFS, and 
has opened a London office. Something 
else people were looking for was Kerberos 
5 support. There were complaints about 
poor support from Transarc, especially 
with the 5 pm EST closing time that is 
inconvenient for West Coast customers. 
KNFS was discussed as something that 
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people were experimenting with and had 
had some success with. In the end, people 
had little faith in most of the alternatives 
to AFS (NFS, DFS, CacheFS) and felt that 
the advantages of AFS outweighed the 
problems they had encountered. 

ADVANCED TOPICS WORKSHOP 

Adam Moskowitz, Facilitator 
Rob Kolstad, Co-chair and Scribe 

Summary by Bruce Alan Wynn 

We first went around the room introduc¬ 
ing ourselves, the quantity of users and 
the quantity and type of hosts we sup¬ 
ported (whether individually or as part of 
a team), and two to five topics we wanted 
to discuss during the day. We came up 
with ranges of up to 10,000 users, 10,000 
PCs, 2,000 Macs, and 3,000 UNIX hosts 
of various flavors. Other notes were mul¬ 
tiple terabytes of disk storage (with pro¬ 
jected short-term growth to exceed a 
petabyte), strange printer requirements, 
and extremely high growth rates (up to 
400% a year). 

We determined that we wanted to talk, in 
general, about: 

■ Consistency/standardization in sysad¬ 
min practices as organizations grow 

■ Cool system-administration tools and 
paradigms 

■ Specific hot technologies/paradigms to 
prepare for/crystal ball 

First, we discussed the issue of internal 
consistency and standardization in tech¬ 
nical practices. We tried to look at the 
“problem,” but realized that we all had 
slightly different ways of looking at it — 
which was not surprising considering 
that we had 31 people in the room, all 
with different backgrounds and experi¬ 
ences. We seemed to agree in general that 
creating standards is challenging, enforc¬ 
ing them is a hard problem, and that 
there are many more variables than may 
be obvious at first look. The concept of a 
“taxonomy” or categorization of prob¬ 


lems into areas seemed to make sense to a 
lot of the folks present. 

We next had a free-form discussion on 
cool system-administration tools and 
paradigms. Some general comments 
were: 

■ DSL is great. (10 people have 56K or 
more to the home, all but one of those 
has 56K or faster bidirectionally; six 
people have >128K. Most of these are 
business-paid and not individual- 
paid.) 

■ MRTG (discussed in Tobias Oetiker’s 
paper at the conference) was hailed as a 
wonderful network-mapping tool. It 
uses SNMP polling on a 5-minute 
interval and creates Web pages with 
usage graphs. It ages data appropriately 
and is freely available. Big Brother, a 
systems monitoring package, integrates 
with MRTG and is also freely available. 

■ Intrusion-detection systems now are in 
the same sort of not-yet-well-under- 
stood position as firewalls were a 
decade ago. While then we had free 
firewalls which later became commer¬ 
cial, now we have commercial intru¬ 
sion-detection systems (IDS) even 
though the problem is neither under¬ 
stood nor solved. 

■ Turnover can be interesting. 16 people 
changed jobs at least once in the past 
year. Four of these were internal (same 
company) job changes. Raises in the 
new job ranged from 0-90% and 
seemed to average around 28%. And 14 
people present have open requisitions 
they are actively hiring for. 

■ Enforcing the use of a PDA like the 
PalmPilot has improved the follow- 
through for members of the group. 
Many folks at one company have 
bought one with their own money. 

■ 19 of those present carry a cell phone; 
26 carry pagers; seven carry authenti¬ 
cation devices. A few have two-way 
pagers; 12 participants pay at least part 


of their monthly fees for the portable 
communications devices. 

■ Five members use a Ricochet or similar 
device for wireless digital communica¬ 
tions. 14 more would use it if it were 
available in their area. 

■ Seven people have an agreement to 
attend conferences annually. In spite of 
that small number, 10 have some kind 
of permission to attend more than two 
per year. A couple can go to even more 
if they have papers presented at them. 
Everyone pretty much gets at least one 
per year. About half can attend two or 
more per year, depending on circum¬ 
stances. 

■ Some cool utilities are ssh and Curl. 
LDAP or similar directory services are 
on the rise; 11 attendees have this. 

■ Cordless phones in machine rooms are 
a major win. 

■ Tools sometimes die for lack of nurtur¬ 
ing. It would be nice if there were some 
way to solve that problem (like a 
MacArthur grant type of thing). Even 
finding current versions is too hard. 

See <ftp.sage-au.org.au> for lots of sysad¬ 
min tools. 

Next we discussed hot technologies, 
rumors, and similar prognostications. 

One hot technology we talked about is 
XML, the Extensible Markup Language. It 
is self-verifying, easy to parse, easy to 
search, and has a universal file format. It’s 
different from SGML in that it doesn’t 
include the hard-to-implement features. 
XML supports Unicode. Unicode is the 
next hot technology we see on the hori¬ 
zon. It represents all characters (including 
nonRoman alphabets like Cyrillic, 
Hebrew, and Farsi). Microsoft Office 
2000 uses Unicode; rumor has it that 
Word already supports it. 

Other predictions are: Voice over IP will 
be a hot technology soon; directory ser¬ 
vices are becoming more important; 
applications will support more location 
independence; voice input and/or recog- 


Februaryl999 ;Iogin: 


17 


CONFERENCE REPORTS 





nition will grow in the next year; and dig¬ 
ital camera use will continue to rise. 

SAGE COMMUNITY MEETING 
AND CANDIDATES FORUM 

Summary by Carolyn M. Hennings 

The annual SAGE Community Meeting 
had a significantly larger attendance this 
year than in the past two years that I’ve 
attended. Pat Wilson kicked off the meet¬ 
ing with announcements, introduced the 
candidates for the upcoming Executive 
Committee elections, and moderated a 
question-and-answer session. 

Announcements included the status of 
publications in the “Short Topics” series. 
Educating and Training Systems 
Administrators is in the mail to SAGE 
members. Pat provided an update on the 
certification debate, saying that an advi¬ 
sory committee has been formed and 
charged with the investigation of whether 
or not SAGE should continue to pursue 
the certification issue and how it should 
be done if SAGE chooses to move for¬ 
ward. Pat mentioned the efforts of the 
“Day-in-the-Life” survey and announced 
that Rob Ferrell and Brian Kirouac have 
been named SAGE Historians. Items to 
watch for are “How-To Notes,” revamped 
Rosetta Stone, new booklets on site audits 
and on hiring system administrators. 
Efforts are under way in reviewing the 
ethics policy and in developing a mentor¬ 
ing program. 

Moving on to the candidates forum, Pat 
asked the candidates to introduce them¬ 
selves, and the floor was opened up for 
questions. Question topics included the 
certification issue, the ethics policy, edu¬ 
cation and training, the inclusion of 
Windows NT topics in the LISA pro¬ 
gram, and the markets view of SAGE. 

In the BPF format, the SAGE 
Community Meeting continued under 
the guidance of Hal Miller, SAGE 
Executive Committee President. The 
open-forum discussion focused on the 


professional development of systems 
administrators in the sense of gaining 
skills to communicate and work with 
management. David Parter, chair for 
LISA ’99, took many suggestions for top¬ 
ics for the next conference. The smaller 
group meeting provided an informal 
opportunity to meet other individuals 
who are actively involved with SAGE at 
the national and local levels. 

CLOSING SESSION 

Summary by Carolyn M. Hennings 

At a lot of conferences the attendee pop¬ 
ulation markedly diminishes on the last 
day of the conference. LISA is different 
for one reason. As always, Rob Kolstads 
LISA Quiz Show is a major attraction at 
the conclusion of the conference. 

This year was no exception. Maintaining 
the same format from previous years and 
always making technical improvements, 


conductors set the stage, but the final 
question was “Who wrote West Side 
Story?” One never knows what to expect. 

A bonus this year was the “Tournament 
of Champions.” Last year Snoopy Beagle, 
who hails from Germany, lost to Hal 
Pomeranz. Snoopy then challenged the 
fairness of the Quiz Show, saying that too 
many questions were based on American 
pop culture. Apparently Rob Kolstad 
heard enough feedback over the past year 
regarding this issue and decided to rectify 
the situation. 

The “Tournament of Champions” was a 
contest between this year s winner, last 
years winner, and Snoopy. Notable cate¬ 
gories for the final round included 
European history, television shows, secu¬ 
rity, and match the dictator. In a surpris¬ 
ing victory, Daniel Boyd made mince¬ 
meat of his competitors. Who knows 
what will happen next year! 



The 1998 Champion, Daniel Boyd, with Rob Kolstad 


the Quiz Show gets better and better. 
Conference attendees vied to answer 
questions in areas such as UNIX admin¬ 
istration, the WWW, computer execu¬ 
tives, physics, circus acts, electrical cur¬ 
rent, coins, certification questions, and 
European dictators. In the category of 
conductors questions about electrical 
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Snoopy 



Dan Geer & Greg Rose comparing ribbons 


Enjoyed the photographs? Attending a USENIX conference? Want to 
contribute to others’ enjoyment? If you take good pictures at the confer¬ 
ence, send them to us. If we can use them in ;login:> though we can’t 
promise you fame or wealth, we do promise to credit you as 
photographer. 
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SAGEnews & features 


( Off to See the Wizards 


by Tina Darmohray 

Tina Darmohray, editor of 
SAGE News & Features, is a 
consultant in the area of 
Internet firewalls and net¬ 
work connections, and fre¬ 
quently gives tutorials on 
those subjects. She was a 
founding member of SAGE. 

<tmd@usenix.org> 


I’m just back from the very successful 
and well-attended LISA conference, 
where the brave attendees (“ Boston in 
December 7 ”) were rewarded with down¬ 
right balmy weather. There was lots to see 
and do, and, most of all, fellow admins to 
talk to; I always learn so much in the 
short time I spend at LISA. The high 
point of the conference for me was 
receiving the SAGE Outstanding 
Achievement Award. Fortunately I wore a 
red turtleneck sweater that morning, 
which helped tone down the matching 
cheek-color I was sporting as I heard my 
name announced. I’m grateful for the 
honor. 

I think recipients of awards like this one 
seldom come to the podium alone. By 
that I mean that they’ve likely been assist¬ 
ed, influenced, mentored, and encour¬ 


aged by a lot of people along the way, and 
therefore represent a whole group of peo¬ 
ple’s effort. I know that’s the case with 
me. Many of the people I worked with 
early in my career were very willing to 
answer questions and point me in the 
direction of additional sources of infor¬ 
mation. 

Later, when my coworkers and other pro¬ 
fessional friends felt I had useful infor¬ 
mation to share, they encouraged me to 
do so by submitting a paper to LISA, 
then editing the Job Descriptions booklet, 
and now editing for ;login:. Each time I 
considered doing any of these I always 
felt ill-qualified. Each time, however, I 
also met willing mentors who shepherded 
me along until I became able to accom¬ 
plish the task on my own. 

I liken my experience to Dorothy and the 
Wizard of Oz. She very much wanted an 
audience with the Wizard, anticipating 
how much he could help her. At first, the 
Wizard seemed unapproachable, and 
Dorothy questioned her qualifications 
and convictions. Again her supportive 
compatriots, who were pivotal during her 
journey to the Emerald City, continued 
to urge her forward, even though the out¬ 
ward image of the Wizard was so daunt¬ 
ing. Luckily, she found that the Wizard 
wasn’t really scary and that he’d also help 
her toward her goal. 



I’d like to encourage all of you to start 
your own journey toward Oz! One way to 
get going is to write an article for ;login:. 
If you’ve got a neat tool, interesting solu¬ 
tion, or great idea to share with the SAGE 
community, we’d love to hear from you. 
Just in case the unknown twists of the 
yellow brick road are what’s keeping you 
from getting started, let me share some 
answers to the most FAQs I get from 
first-time ;login: contributors. 

Format: Since we do the formatting, all 
we need are ASCII submissions. 

Length: Theoretically, there’s no mini¬ 
mum or maximum length for an article. 
Something chapter-length might be a lit¬ 
tle long, but we can always work with you 
to split it across more than one issue. 
(Hey, these are good problems to have!) 

Deadlines: Our publishing deadline rolls 
around every other month, usually dur¬ 
ing the first week (check the back cover 
of a recent issue), but you can submit 
something at any time. If you don’t know 
if your topic would be suitable, you can 
run it past us ahead of time and we’ll let 
you know up front, before you’ve sunk a 
lot of effort into polishing your prose. 

Help: Most important, if you’ve got the 
topic, but share the common, self-per¬ 
ceived, “I can’t write” syndrome (which 
I’ve certainly ascribed to myself), we’ll 
help you with that. We have bona fide 


SAGE, the System Administrators Guild, is a 
Special Technical Group within USENIX. It is 
organized to advance the status of computer 
system administration as a profession, 
establish standards of professional excellence 
and recognize those who attain them, develop 
guidelines for improving the technical and man¬ 
agerial capabilities of members of the 
profession, and promote activities that advance 
the state of the art or the community. 

All system administrators benefit from the 
advancement and growing credibility of the pro¬ 
fession. Joining SAGE allows individuals and 
organizations to contribute to the community of 
system administrators and the professions as a 
whole. 


SAGE membership includes USENIX membership. 
SAGE members receive all USENIX member bene¬ 
fits plus others exclusive to SAGE. 

SAGE members save when registering for USENIX 
conferences and conferences co-sponsored by 
SAGE. 

SAGE publishes a series of practical booklets. 
SAGE members receive a free copy of each book¬ 
let published during their membership term. 

SAGE sponsors an annual survey of sysadmin 
salaries collated with job responsibilities. 

Results are available to members online. 

The SAGE Web site offers a members-only Jobs- 
Offered and Positions-Sought Job Center. 


SAGE STG EXECUTIVE COMMITTEE 

President: 

Hal Miller <halm@usenix.org> 

Secretary: 

Tim Gassaway <gassaway@usenix.org> 

Treasurer: 

Barb Dijker <barb@usenix.org> 

Members: 

Helen Harrison <helen@usenix.org> 

Amy Kreiling <amy@usenix.org> 

Kim Trudel <kim@usenix.org> 

Pat Wilson <paw@usenix.org> 
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wizards of editing on staff for just that 
reason. And you should know that every¬ 
thing in ;login: gets edited to some degree, 
so you really, really are not going to stand 
out, or have any reason to be embar¬ 
rassed, if your article needs some editori¬ 
al assistance. In fact, consider any editing 
that is done as free training on how to 
improve your writing style. I can state 
from experience that the editing process 
has been the single most effective tool in 
bettering my own writing skills. 

Topics: Finally, if you like to write but are 
in search of a topic, we brainstormed on 
some of the neat topics and additional 
items we’d like to feature in ;login: in the 
upcoming year. They include: 

Webmastering 

security 

NT integration 

64-bit programming 

open source 

production (email) servers 

most interesting Web sites 

most useful tools 

USENIX community news 

If you are working on any of these or are 
interested in writing about them, please 
get in touch with us. 


So, if you’re inclined, get started on con¬ 
tributing to ;Iogiti:. I can promise you’ll 
get mentoring and encouragement, and 
you’ll enjoy the wizards you meet in the 
process! 


by Hal Miller A 

Hal Miller is president of the SAGE STG Executive 
Committee. 

<halm@usenix.org> 


As I complete the intrusion-recovery 
checklist for a series of break-ins, I sit 
here contemplating next steps for SAGE. 

I am now at a university, in an environ¬ 
ment very different from those I’ve been 
in for most of my career. In the past I 
have “owned” the network, servers, and 
clients. If any user had root access (or 
equivalent), the machine was on a sepa¬ 
rate firewalled segment, or I worked 
closely enough to trust the individual. 
Now, I have no access to the wiring clos¬ 
ets nor their contents, except as a general 
user of network services. I have control 
over only a small number of machines, 
and access of any kind (often excluding 
root) to only a small percentage of the 
machines on my net. There are no pro¬ 
mulgated policies; my docs are still sitting 
on people’s desks for review. 


Three machines were compromised this 
week, all by the same attack method, 
although by two unrelated groups of 
attackers. I had no access to any of those 
machines, and in fact wasn’t even aware 
of the existence of one of them (installed 
on the network by the owner and the 
university’s networking support group). 
The attackers in both cases also tried 
machines my group does control, all of 
which fended off the attacks and reported 
to us. We observed many other attacks, 
but these used “old” methods and failed 
to gain access to those machines logging 
the attempts. 

Suddenly we were heroes. We went in and 
collected legal evidence, proving to the 
machine owners the extent of damage 
done, then rebuilt the boxes per industry 
standard. We now have “shared” control, 
which probably means the users do what¬ 
ever they desire, and we get the blame 
when that causes a problem. 

What exactly are the issues here, and how 
can SAGE help? 

■ How do we get policies implemented? 

■ What do we do if we can’t get policies 
implemented, or in the interim while 
review is under way? 

■ How should we deal with responsibili¬ 
ties split between system and network 
admins, especially when they don’t 
agree on requirements? 


Extending Our 
.Goal Set 


SAGE MEMBERSHIP 
<office@usenix.org> 

SAGE ONLINE SERVICES 

Email server: <majordomo@usenix.org> 
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SAGE SUPPORTING MEMBERS 

Atlantic Systems Group 

Collective Technologies 

Compaq Computer Corporation 

Deer Run Associates 

D.E. Shaw & Co. 

ESM Services, Inc. 
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■ How should we deal with responsibili¬ 
ties split between system administra¬ 
tors and user-owners who have 
root/administrator access? 

■ How do we deal with requirements for 
sharing facilities/services on our own 
networks to untrusted boxes, let alone 
remote nets? 

I have no immediate authoritative and 
conclusive answers. I do think, however, 
that SAGE is the right place to establish 
those conclusive answers. Many of us are 
faced with similar situations, and these 
problems affect the industry as a whole. 
They are among the kinds of problems 
SAGE should tackle. 

How? Well, thus far SAGE has been con¬ 
centrating on bootstrapping itself. 
Current emphasis is on the advancement 
of the individual sysadmin through cre¬ 
ation and operation of conferences and 
workshops, ethics, certification and edu¬ 
cational projects. It is time to start look¬ 
ing at what we can do for the industry in 
addition to supplying qualified sysad¬ 
mins. The Short Topics booklet, System 
Security: A Management Perspective y is a 
start in this arena, but we need to plan 
out a strategy for far more and bigger 
goals. What might those goals be? Some 
possibilities: 

■ Raising the security standard of the 
average site so high that “casual” crack¬ 
ing ceases to be of interest 


■ Raising the consciousness level of the 
average site’s management regarding 
the requirement for an appropriate 
level of applied system administration 
resource and security effort 

■ Creating a trusted place that organiza¬ 
tions or individual sysadmins might 
turn for immediate assistance for any 
form of sysadmin problem, and mak¬ 
ing it known and accepted 

■ Establishing minimum “standards of 
practice” guidelines - a tricky and 
controversial area 

We have made some progress already, 
such as Short Topics booklet A Guide to 
Developing Computing Policy Documents , 
but we are still in the very early stages of 
addressing the scope of problems faced 
by the “average” sysadmin. Our approach 
so far has been to provide an overview of 
the methods we might apply to the 
“black art” of system administration, and 
to assume that local sysadmins will know 
how to follow through. Perhaps most of 
us old-timers can do this, perhaps not. 
Can newer folks do so, or are they in 
need of more concrete examples? The 
new “How-To Notes” series in ;login: 
should address this to some degree, but is 
that enough? It seems that uncountable 
sites out there have site sysadmins who 
are not SAGE members. If they don’t take 
our soon-to-be education programs, will 
they be needing checklists? Is it right that 


we provide such a thing? I’d like to see 
some discussion on these questions. 
Don’t hesitate to write for ;login: y to 
<sage-members>, or directly to me. 

By the time you read this, another elec¬ 
tion will be behind us, and a new 
Executive Committee will be setting new 
directions. While they do so, it is not too 
early to begin thinking about what you 
might wish to do to prepare for the com¬ 
mittee that will follow them and how to 
get those projects started. It might even 
be too late in some cases, Other organi¬ 
zations are attempting to make, codify, 
or otherwise influence our profession 
from their own viewpoints, typically 
profit-centered rather than sysadmin- 
centered. We can’t drop the ball for a 
moment, especially at this point in our 
development. 
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how-to 


[install Anonymous FTP __ 

This note describes how to configure an anonymous FTP server on a UNIX-like 
operating system. The steps for configuring the freely available Washington 
University FTP daemon software (WU FTP) are described, but administrators 
wishing to use their vendor's anonymous FTP server may do so simply by skip¬ 
ping Section 1, Section 2.7, Section 3.1 (though see Section 3.1.2 for guide¬ 
lines on how to create an /etc/ftpusers file), and Section 3.3. 

The WU FTP software should be used if the administrator wishes to allow file uploads 
in a secure fashion (Section 2.7). In particular, a good set of requirements for file 
upload areas includes 

I. No user may get a directory listing in the upload area or any of its subdirectories 
(helps prevent disclosure of proprietary information in file or directory names). 

II. Files may only be uploaded into subdirectories of the upload area (helps 
protect -ftp/ (bin, lib,dev,etc,pub} ). 

III. Files that have been uploaded may not be downloaded by any anonymous user 
(prevents your site from being used as a “warez” site for the dissemination of copy¬ 
righted materials). 

IV. Files that have been uploaded may not be overwritten (helps prevent confusion and 
trojan-horse attacks). 

Requirement I plays havoc with graphically oriented, “point-and-click” FTP clients that 
insist on getting directory listings in order to function properly. Most of these clients 
allow the user to input pathnames from the keyboard, but some amount of user educa¬ 
tion is likely to be required. 

Requirements III and IV are enforced by configuring the WU FTP daemon to make all 
uploaded files be read-only (mode 400) owned by root. This implies an administrator 
or some external process that retrieves files from the upload area and makes them avail¬ 
able to internal users. 

1. WU-ftpd 

The Washington University FTP daemon is the de facto standard FTP server for anony¬ 
mous FTP sites. This is primarily due to the wide variety of features it supports - fea¬ 
tures that make the code more difficult to audit and therefore more dangerous from a 
security perspective. If you only need to allow “vanilla” anonymous FTP access (no file 
uploads, no nonanonymous access), then skip this section and use your OS vendor’s 
ftpd. 

7.7. Download 

1.1.1 Connect to ftp://ftp.academ.com/pub/wu-ftpd/private/ 

1.1.2 Download latest beta release (currently wu-ftpd-2.4.2-beta-18 . tar.Z) 

1.1.3 Unpack in appropriate source directory 

# mv wu-ftpd-2.4.2-beta-<vers>.tar.Z <srcdir> 

# cd <srcdir> 

# zcat wu-ftpd-2.4.2-beta-<vers>.tar.Z I tar xf - 

where <vers> is the appropriate version number (see previous step) and <srcdir> is 



<hal@deer-run.com> 


by Hal Pomeranz 

Hal Pomeranz is the Chief 
Operations Architect for 
Corio, Inc., an Application 
Services Provider based in 
Redwood City, CA. 


ANNOUNCING THE SAGE 
"HOW-TO NOTES" SERIES 

With the accompanying first entry by Hal 
Pomeranz, SAGE is pleased to announce the com¬ 
mencement of the How-To Notes series. Based on 
an idea floated in last year's <sage-member> 
survey, this series is a collection of basic infor¬ 
mation intended to assist sysadmins in quickly 
getting something running. They will cover such 
points as where to obtain software, what hard¬ 
ware may be required, and what steps are neces¬ 
sary to get a minimal service configured for 
“average use.” The notes are not designed for 
advanced configurations, complex installations, 
or specialized situations. All will eventually be 
available from the SAGE Web pages. The series 
editor (not yet selected) will manage new topics 
and a review cycle. 

Questions, comments, and suggestions are 
always welcome. We hope this series proves valu¬ 
able to the membership. 
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The most important and 
most difficult part of set¬ 
ting up any anonymous 
FTP installation is getting 
the directory permissions 
right. Errors in this step 
can allow outsiders to 
modify or overwrite files in 
your anonymous FTP area 
and possibly gain shell 
access to your machine. 


some directory on your system where you keep third-party source code. 

1.2. Build Process 

1.2.1 Move to configuration directory. 

# cd wu-ftpd-2.4.2-beta-<vers>/src/config 

1.2.2 Edit appropriate configuration file for your OS (consult .. / .. / INSTALL for OS 
type information). If USE_ETC is set, replace this with USE_ETC_FTPD. 

1.2.3 Return to top-level source directory. 

# cd ../.. 

1.2.4 Initiate build process. 

# sh build CC=<comp> <target> 

where <comp> is your C compiler (cc by default) and <target> is the appropriate OS 
identifier. 

1.3. Install 

1.3.1 Edit top-level Makefile and set BINDIR, ETCDIR, and MANDIR appropriately. 
WARNING! On most architectures, the default values for these variables will overwrite 
existing OS binaries and other files. This is probably not what you want. 

1.3.2 Trigger install process. 

# make install 

2. Create anonymous FTP directory 

When a user logs in as anonymous, FTP daemons call chroot () to restrict that users 
access only to files in your anonymous FTP area. Make sure that your anonymous FTP 
area lives in its own partition and don’t make any symbolic links into or out of this 
area. Also note that the most important and most difficult part of setting up any 
anonymous FTP installation is getting the directory permissions right. Errors in this 
step can allow outsiders to modify or overwrite files in your anonymous FTP area and 
possibly gain shell access to your machine. 

2.1. Create top-level FTP directory 

2.1.1 Make directory. 

# mkdir <path> 

where <path> is where the anonymous FTP area is rooted (e.g., /usr/local/ftp, 
/export/ftp, etc.). 

NOTE: This step may not be required if the anonymous FTP area is going to be located 
on an already-mounted partition, or you may have to modify your fstab file and mount 
the FTP area by hand before proceeding. 

2.1.2 Set ownership and permissions. 

# chmod 755 <path> 

# chown root <path> 

# chgrp root <path> 
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dribble volume manager configuration 


cltXdY (get X,Y below) 


0,0 ROOTDISK 

2,0 ROOTMIRROR 

4,0 SPARE 

0,1 usr 1 oc al/loc al_ra i dlog 

2,1 elib/elib_raidlog 

4,1 depoi/ciopotO 1 

0,2 elib/elibOl 

2,2 elib/elib03 

4,2 elib/elib05 

0,3 

2,3 

4,3 

0,4 

2,4 

4,4 

1,0 usrlocal/localOl 

3,0 u,srlocal/local02 

5,0 usrlocal/local03 

1,1 elib/elib07 

3,1 elib/elib08 

5,1 elib/elib09 

1,2 elib/elib02 

3,2 elib/e!ib04 

5,2 elib/elib06 

1,3 

3,3 

5,3 

1,4 

3,4 

5,4 


RAID-5 volumes: usrlocal, elib (these are not mirrored but do have RAID5 parity info) 

Simple volumes: depot (not mirrored, no parity info!) 


Boot device: 

/io-unit@f,e0200000/sbi@0,0/SUNW,soc@2,0/SUNW,pln@a0000000,78d343/SUNW,ssd@0,0:a 


Boot mirror: 

/io-unit@f,e0200000/sbi@0,0/SUNW,soc@2,0/SUNW,pln@a0000000,78d343/SUNW,ssd@2,0:a 

































2.1.3 Move to new directory. 
# cd <path> 


2.2. Install is program. 

2.2.1 Make ~f tp/bin directory. 
# mkdir bin 


2.2.2 Copy binary. 


# cp /bin/Is bin 

2.2.3 Set ownership and permissions. 

# chmod 111 bin bin/Is 

# chown root bin bin/Is 

# chgrp root bin bin/Is 


2.3 Install shared libraries. 

NOTE: Commands below are appropriate for Solaris systems. Consult OS documen¬ 
tation to find out appropriate libraries for other systems. Skip this step if you can build 
a statically linked Is program on your machine (Is sources available from any GNU 
archive in the fileutils package). 

2.3.1 Make ~f tp/lib directory. 

# mkdir -p usr/lib 

2.3.2 Copy libraries. 

# for lib in ld.so.l libc.so.l libdl.so.l libintl.so.l \ 

> libw.so.l nss_files.so.l 

> do 

> cp /usr/lib/$lib usr/lib/$lib 

> chmod 555 usr/lib/$lib 

> done 

2.3.3 Set ownership and permissions. 

# chmod 111 usr usr/lib 

# chown root usr usr/lib usr/lib/* 

# chgrp root usr usr/lib usr/lib/* 

2.4. Make devices. 

NOTE: The arguments for mknod shown here are for Solaris systems. For other 
machines, copy the major/minor device numbers, ownership, and permissions from 
/dev (the Is -1 command may be used to show major/minor device numbers). 

2.4.1 Create -ftp/dev. 

# mkdir dev 

2.4.2 Create device files. 

# mknod dev/tcp c 11 42 

# mknod dev/zero c 13 12 

2.4.3 Set ownership and permissions. 

# chmod 111 dev 

# chmod 666 dev/tcp 
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Make sure that all of the 
files and subdirectories in 
the ~ftp/pub directory 
are owned by root or 
somebody other than the 
anonymous user, ftp, so 
that there is no chance of 
the files being overwritten. 


# chmod 444 dev/zero 

# chown root dev dev/top dev/zero 

# chgrp sys dev dev/top dev/zero 

NOTE: The permissions for dev/top shown above are correct. 

2.5. Configure -ftp/etc directory. 

2.5.1 Create -ftp/etc directory. 

# mkdir etc 

2.5.2 Create ~f tp/etc/passwd file as follows: 

root:x:0:1:0000-Admin(0000):/:/sbin/sh 
ftp:x:90:1:Anon FTP User:/:/sbin/sh 

WARNING! Do not put actual passwords in the ~f tp/etc/passwd file! Do not create 
-ftp/etc/shadow or similar file! 

2.5.3 Create -ftp/etc/group file. 

root:*:0: 
other::1: 

2.5.4 Create -ftp/etc/nsswitch.conf file (not required on nonSolaris machines). 

passwd: files 
group: files 

2.5.5 Create the welcome.msg, message. toomany, and pathmsg files in -ftp/etc as 
documented in Appendices B-D. 

2.5.6 Set ownership and permissions. 

# chmod 444 etc/* 

# chmod 111 etc 

# chown root etc etc/* 

# chgrp root etc etc/* 

2.6. Make download area. 

NOTE: Files in this area will be readable by anonymous users. Make sure that all of the 
files and subdirectories in this directory are owned by root or somebody other than the 
anonymous user, ftp, so that there is no chance of the files being overwritten. 

2.6.1 Make -ftp/pub directory. 

# mkdir pub 

2.6.2 Set ownership and permissions. 

# chmod 555 pub 

# chown root pub 

# chgrp root pub 

2. 7 . Make upload directories. 

NOTE: Skip this section if you do not wish to allow uploads to your server at all. Do 
not allow uploads if not using WU FTR 

2.7.1 Make -ftp/incoming directory. 

# mkdir incoming 
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2.7.2 Set ownership and permissions. 

# chmod 111 incoming 

# chown root incoming 

# chgrp root incoming 

2.7.3 Create one or more individual upload areas (substitute some directory name for 
<dir> below). 

# mkdir incoming/<dir> 

# chmod 311 incoming/<dir> 

# chown <uid> incoming/<dir> 

# chgrp root incoming/<dir> 

where <uid> is the numeric UID you will use for the ftp user in the system password 
file. 

3. Other system-configuration tasks 

With the anonymous FTP area successfully configured, the WU FTP software itself 
must be configured, password entries created, and modifications made to inetd.conf. 

3. 7. Create WU FTP configuration files. 

NOTE: Skip this section if not using WU FTP, but do create a file called 
/etc/ftpusers per Section 3.1.2 (but not /etc/f tpd/ftpusers as documented 
there). 

3.1.1 Make /etc/ftpd directory 

# mkdir /etc/ftpd 

# chmod 700 /etc/ftpd 

# chown root /etc/ftpd 

# chgrp root /etc/ftpd 

3.1.2 Create /etc/f tpd/ftpusers file. This file contains the list of all users (one per 
line) not allowed to FTP into the server. It should include root, daemon, bin, nobody, 
and similar accounts which are not associated with real users. 

3.1.3 Make symlink from traditional /etc/ftpusers location. 

# In -s /etc/ftpd/ftpusers /etc/ftpusers 

NOTE: If /etc/ftpusers already exists, make sure that /etc/ftpd/ftpusers is a 
superset of this file, then remove /etc/ftpusers and create the symlink. 

3.1.4 Make empty ftpconversions file. 

# touch /etc/ftpd/ftpconversions 

3.1.5 Create f tpaccess configuration file from Appendix A. 

3.1.6 Set ownership and permissions. 

# chmod 600 /etc/ftpd/ftp* 

# chown root /etc/ftpd/ftp* 

# chgrp root /etc/ftpd/ftp* 

3.2. Make password entry. 

3.2.1 Create entry for the ftp user in /etc/passwd. 

ftp:x:<uid>:<gid>:Anonymous FTP Account:<path>:/dev/null 


Create 


/etc/ftpd/ftpusers file. 


This file contains the list 
of all users (one per line) 
not allowed to FTP into the 
server. It should include 



root, daemon, bin, 


nobody, and similar 
accounts which are not 
associated with real users. 
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Do not use a valid DES 
password string for the 
ftp user. Instead use a 
string like * or lock or 
*np*. Do not use an 
empty password field for 
the ftp user! 


<uid> is the numeric user ID used in Section 2.7.3 and <gid> is some unused numeric 
group ID. <path> is the root of the anonymous FTP area as used in Section 2.1. 

3.2.2 Create corresponding entry in /etc/shadow or other file as required by your 
operating system. Do not use a valid DES password string for the ftp user. Instead use 
a string like * or LOCK or *np*. Do not use an empty password field for the ftp user! 

3.3. Make inetd changes. 

NOTE: Skip this section if not using WU FTP. 

3.3.1 Edit /etc/inetd.conf (on some systems, /etc/inet/inetd.conf) and replace 
the existing entry for ftp with ftp stream tcp nowait root <BINDIR>/ftpd ftpd 
where <BINDIR> is the directory chosen for the install of WU FTP binary in Section 
1.3.1. 

3.3.2 Get process ID of running inetd using ps and grep. 

3.3.3 Send SIGHUP to running inetd 

# kill -HUP <pid> 

where <pid> is the process ID determined in the previous section. 

Appendix A. WU FTP f tpaccess Configuration File 

# real users may log in from the 172.16.0.0/16 network— you 

# probably want to use your internal network address space here 

# instead. Anonymous users may log in from any IP address. 

# 

class users real 172.16.0.0 localhost 
class anon anonymous * 

# Three login attempts are permitted before the user is dropped and 

# a message is logged via syslog. 

# 

loginfails 3 

# Only ten real users are allowed at any time. 100 anonymous users 

# are allowed any Saturday or Sunday, or any weekday between 6pm 

# and 6am local time. At other times, only 60 anonymous users are 

# allowed. 

# 

# Obviously, you'll need to tune these parameters for your server 

# and bandwidth usage limits... 

# 

limit users 10 Any /etc/msg.toomany 

limit anon 100 SaSuIAnyl800-0600 /etc/msg.toomany 

limit anon 60 Any /etc/msg. toomany 

# The first two lines cause any files named README* to be printed 

# if the user logs in (in this case the README* files must be in 

# -ftp) or if the user changes directory into a subdirectory 

# containing a README* file. 

# 

# The next two lines cause the file ~ftp/etc/welcome.msg to be 

# displayed on each login and ".message" files to be displayed if 

# the user "cd"s into a directory containing such a file. 

# 

# "readme" files and "message" files are identical except "message 1 

# files are only displayed once per user session, whereas "readme" 

# files are displayed every time the given condition is met.# 
readme README* login 

readme README* cwd=* 
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message /etc/welcome.msg login 
message .message cwd=* 

# Anonymous users are prompted to enter their email address as a 

# password. 

# "passwd-check" prints a "warn"ing message if the password doesn't 

# look like a standard ("rfc822") email address. If you choose 

# "enforce" rather than "warn" you will end up denying access to 

# many, many people on the Internet who really don't know what 

# their email address is... 

# 

passwd-check rfc822 warn 

# Stop anonymous users from using "interesting" FTP commands. 

# Directory permissions in -ftp should also stop this behavior, 

# but a little strength in depth never hurts ... 

# 

delete no anonymous 
overwrite no anonymous 
rename no anonymous 
chmod no anonymous 
umask no anonymous 

# NOTE: Insert the correct pathnames for your -ftp directory 

# here!!! Uploads are generally not allowed in -ftp, but are 

# allowed in subdirectories of -ftp/incoming. Files uploaded here 

# will end up being owned by root, mode 4000 so that they cannot 

# be overwritten by other anonymous users. 

# 

upload /local/ftp * no 

upload /local/ftp /incoming/* yes root root 0400 

# Allow anonymous users to specify filenames containing letters, 

# numbers, dash (* — '), underscore and period (' . ') but not 

# paths which begin with period (e.g., '../../../../../../. ./etc/passwd', 

# which shouldn't work anyway due to chrootO) or dash (could be 

# trying to play tricks with STDIN/STDOUT). 

# 

path-filter anonymous /etc/pathmsg A [-A-Za-z0-9_\.]*$ A \. A - 

# Log all commands issued by real users as an audit log. Log all 

# transfers to or from this server by real or anonymous users. 

# 

log commands real 

log transfers anonymous,real inbound,outbound 

# Email address used for %E in message files 

# 

email admins@sysiphus.com 

Appendix B. Welcome Message (~ftp/etc/weicome.msg) 

This is the anonymous FTP server for <insert your company name 
here>. 

It is primarily for the use of the customers and employees of <your 
company name>. Please do not abuse this resource. Thank you for 
your cooperation. 

Note that file transfers to and from this server are now logged. If 
this bothers you, please log off now. 

If you have any questions about this server, and especially if you 
have any problems using it, please contact: 

Some Body somebody@yourdomain.com 
Some Title +1 (666) 555-5555 
Your Company Name 

If your ftp client has problems with receiving files from this 
server, send a as the first character of your password (e-mail 
address) . 
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<jsellens@uunet.ca> 


Appendix C. Too Many Users Message (~ftp/etc/msg. toomany) 

Sorry, there are too many users on this server at the moment. 

Please try again in 15 minutes or so. 

If you believe you have received this message in error, please send 
email to %E. 

Appendix D. Invalid Pathname Message (~ ftp/etc/pathmsg) 

Please choose a pathname that does not contain special characters. 


on reliability 


by John Sellens 

John Sellens has recently 
joined the Network 
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Canada in Toronto after 11 
years as a system adminis¬ 
trator and project leader at 
the University of Waterloo. 


(You and Your Users _) 

This time around, let’s discuss user community interaction and how it relates 
to reliability. Recall that we’re providing services, and that users are, of course, 
the reason we provide those services. How can we use our user interaction to 
improve reliability, communicate that increased reliability is one of our goals, 
and help our users become part of the reliability equation? (I’ll mention that 
I'm writing this on an airplane, and as a user of the airline's services, I very 
much want to know that they care about reliability, and I'm more than willing 
to do my part.) Let’s look at the question three ways: communication to users; 
communication from users; and education, training, and publications. 

Communication to Users 

A long time ago I learned that in a number of situations it is not sufficient simply to 
do your job - you must also be seen to be doing your job. By that I mean that some¬ 
times you must be obvious about what you are doing (and why) while you’re doing it. 
Consider, for example, a security guard - the fact of being visible in itself can act as a 
deterrent and reduce the likelihood of “an incident.” 

Reliable system administration is another one of those tasks that are enhanced by visi¬ 
bility. For example, if a system is obviously being run in an organized and disciplined 
manner, are users more likely to act that way themselves, and thereby bring us closer to 
our goals? How can we be obvious about what we’re doing, and why, and how can 
users help? A documented, predictable computing environment will be thought of as 
far more reliable than it would otherwise. 

The standard method for successful presentations is to tell your audience what you’re 
about to tell them, tell them, and then tell them what you’ve told them. There’s a con¬ 
venient parallel for communication to your user community: 

■ Tell them what you expect to do for them and what you expect from them. 

■ Follow through and work toward your commitments, keeping them informed of 
your progress (or lack thereof) and of any failures or incidents that might occur. 
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■ Provide statistics, incident reports, and plans for improvement as you progress. 

Tell Ya What I’m Gonna Do 

Advance communications are often going to be the largest component of your “for¬ 
mal” user communication and are likely to tend toward the “static” rather than the 
“dynamic” You should outline (in greater or lesser detail as your situation demands) 
what services you will (plan to) provide - for example, centralized file service, printing, 
user consulting, and authentication services. The list of services will presumably have 
been arrived at through a process of user consultation, executive fiat, or divine inspira¬ 
tion (or a combination of all three), tempered by your experience and expertise and 
suggestions on what might be most appropriate and useful in your organization. These 
are your “service offerings.” 

Next, you should document your goals for performance and availability, both in terms 
of machine and network performance and availability, and in terms of guaranteed 
response and repair times. These are your “service-level agreements.” For machines and 
networks, you would typically look to such metrics as percentage uptime, response 
times, and network latency guarantees. For example, you could state that your network 
file server will be unavailable less than an hour a month (99.9% uptime), that round- 
trip packet times between major points on your network will be less than 10ms, or that 
there will always be at least 5GB of available disk space (e.g., for image processing). For 
response and repair times, some examples might be next-business-day response for 
installing new personal network connections, accounts created within three hours, file 
restores within eight hours, first response to problem reports within two hours, etc. 
The important thing is to make sure that your goals and guarantees are aligned with 
the business needs of your organization. Figure out what services and activities are 
most important to your users and then determine how you can organize your 
resources (human and machine) to best balance those needs in delivering your ser¬ 
vices. 

The last component in setting expectations is policies and practices. Reasons to estab¬ 
lish service and usage policies include: 

■ conformance with, and the ability to serve, organizational goals (e.g., reserving CPU 
capacity for product developers working to get the next release out the door) 

■ making it possible to meet service-level agreements (e.g., reserving two hours on 
Sunday so that maintenance doesn’t interrupt activities during the week) 

■ ensuring that users aren’t interfering with services to others (e.g., no Quake during 
office hours) 

As a service provider, you will be better off if you proactively define the policies and 
practices you will use to deliver your services. That is, it’s better to define your own 
goals before some other less “reasonable” goals are imposed on you. You will typically 
want to consider such things as: 

■ standard change/maintenance windows (e.g., Saturday mornings, Tuesday nights 
from 11:00 pm until 3:00 am, etc.) 

■ emergency repair procedures, and what constitutes an “emergency” 

■ standard operating hours for services such as the help desk and hardware repairs 

■ a policy for off-hours response and a method of deciding what can wait and what 
has to be fixed immediately 


Recall that we’re providing 
services, and that users 
are, of course, the reason 


we provide those services. 
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u a method for responding to unexpected “incidents,” security-related and otherwise 

■ escalation procedures in case goals aren’t met or problems are more substantial than 
they first appeared 

You will also probably want to outline possible remedies or repercussions for those 
times when you fail to meet your stated goals and guarantees. For an internal service 
organization, these are more likely to involve public “humiliation,” bad performance 
reviews, or lousy parking spots in the company lot. For an external service provider 
(e.g., an ISP or a computing service bureau), the result of a failure to meet the goals ai 
guarantees is likely to involve money. 

Finally, you will need to outline the policies that your user community is expected to 
follow. These policies contribute to reliability by (we hope) freeing you from having tc 
deal with malicious or unexpected acts by your users and establishing a base of under 
standing between users and service providers. The better the understanding between t 
two groups, the easier it is to provide a reliable and understood service. You will likely 
want to cover such areas as: 

■ security, password sharing, snooping, sniffing, hacking, etc. 

■ virus protection, and under what circumstances it is acceptable to add software to 
your systems and networks. 

■ protection of the organization’s equipment (e.g., no latte with more than three suga 
allowed within two feet of a keyboard) 

■ disk-space limits, CPU hogging, and other resource-consumptive areas of contentio 

■ what may or may not be connected to the network, and where (inside or outside th< 
firewall, etc.) 

Change , Status, and Failure Reports 

These reports should be part of ongoing, day-to-day communications with your user 
community. The two most important attributes of these reports are timeliness and 
completeness - you need to provide the information that your users want, need, or 
deserve at the most appropriate time. 

Change reports outline a planned change to your systems or networks and its impact 
(or, even better, lack of impact) on your users, and summarize or confirm its imple¬ 
mentation. These reports are important so that your users can plan their activities 
around expected service disruptions, and so that they can understand and prepare fo 
changes to software or interfaces. Reports would typically outline the expected date ai 
time of the change, its impact, and how to obtain additional information. Internally, 
you should also document (in advance) how to implement that change, how to test tt 
implementation, and, most important, how to back out of the change if necessary 
Change reports should be issued far enough in advance that users have adequate pro; 
ration and warning time, but not so far in advance that they are forgotten. 

Status reports are the ongoing mechanism by which your services can be measured, 
both for trend analysis and to allow for evaluation against your service-level agree¬ 
ments. You would typically track as many of your defined metrics as possible in log 
files, graphs, printed reports, Web pages, summary numbers, and so forth. I recomme 
MRTG[1] as a terrific way to graphically track virtually anything against the calendar 
network traffic, uptime, users, disk space, news or mail traffic, routing-table size, ni n 
bers of outstanding trouble tickets, and on and on. If you’ve got the time and you c ir 
automate the collection and reporting (and it won’t adversely affect your systems), r; 
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as many metrics as you can think of, even if you don’t publish them all. The more infor¬ 
mation you have, the easier your troubleshooting and capacity planning will be. It’s 
common in customer service organizations to track various service metrics (time to res¬ 
olution, phone queue time, calls per hour, etc.). Many (most?) system administration 
organizations could benefit from more proactive status reporting. 

Failure reports are, obviously, something that we would all like to keep to a minimum. 
They are the method by which you report system or network “incidents” to your cus¬ 
tomers, and (ideally) document your plans to ensure that such failures are reduced or 
eliminated in the future. They can also serve as a symbolic way for you to “take respon¬ 
sibility” for an outage and demonstrate your commitment to improving your service. 
Don’t feel you need to wait until an outage is resolved to issue a failure report; your 
users will appreciate your openness and consideration and will also be less likely to con¬ 
tact the help desk about expected up times if you’ve already posted bulletins. The latter 
can be quite useful if you’re with a small organization and the people responsible for 
answering the phones are the same people who are busily trying to fix the problem. 


Track as many metrics as 
you can think of, even if 
you don’t publish them all. 
The more information you 
have, the easier your 
troubleshooting and 
capacity planning will be. 


Statistics, History, and Revisiting the Future 

This is where analysis and prediction come into play. Using the information that you 
have gathered - along with projected changes in usage, information on new projects, 
and normal usage increases - you can generate current and historical statistics, identify 
trends, and predict the future. And when you’ve done it once, you will also be able to 
use it to repredict the future and compare your predictions to what actually happened. 
The capacity-planning uses of this information are obvious, and these reports are also a 
good way to demonstrate your professionalism to your users (and your management!). 


Communication Initiated by Users 

In order to be able to correct problems, deal with “incidents,” and enable some ongoing 
improvement of your processes, you need to provide ways for your user community to 
get you the information you need (and the information that they need you to have). 

I’ve defined this communication as user-initiated in an attempt to gently remind you 
that communication must be two-way - you (of course) have a duty to respond (prefer¬ 
ably in a timely and effective way). 

It typically makes sense to think of two sets of user-initiated communication: 

■ the ongoing day-to-day problem reports, help requests, and requests for enhancement 
(and, if you’re very good, thank you notes for a job well done) 

■ the periodic status reports, general reviews, direction or policy guidelines, and overall 
satisfaction indicators that help determine your direction and activities 

The first can be expected to come from almost anyone within your organization (and 
sometimes from outside your organization too), while the second are more likely to 
come from organizational management, steering committees, user groups, and the like, 
as well as your own surveys and inquiries. 

The most common communication method is (of course) the telephone. Most of us 
have probably experienced those calls out of the blue reporting some major problem, 
with an expectation that we will drop everything and solve it immediately. But as I 
review the communication process, I’ll also review the alternative communication 
methods that you should consider and/or support. 

Let’s assume that you have some number of people responsible for dealing with user or 
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customer queries (the “help desk”), and divide the communication process into three 
stages: the request, tracking and resolution, and your response. (These three stages 
apply to both sets of user-initiated communication, but you will of course adjust your 
reaction and process depending on the type of communication.) 

The Request 

When someone needs help, or wishes to report a problem or request an enhancement, 
they need some “reporting method.” Consider these alternatives: 

Phone: a “well-known” generic problem-reporting phone number, with responsibility 
for answering it distributed among your help-desk staff in some reasonable fashion 
(queue, dedicated “on-call” person, round robin, whoever isn’t busy, etc.). It is, of 
course, convenient if there is mnemonic value to the number (e.g., company extension 
4357 - HELP), but the important things are that it exists and it’s publicized. Note that 
some telephone systems can provide call statistics for your help-desk calls, which is 
handy when you’re trying to prove that you need more staff. 

Email: a well-known email alias, such as <help@company.com> - most of the same consid¬ 
erations apply here as for telephone contacts. 

Newsgroup postings: a local newsgroup used to report problems or request enhance¬ 
ments in larger environments. (I’ve seen this used in a university.) This can cause the 
users to feel a little bit like they’re sending their problem floating off like a message in a 
bottle, but with prompt response and tracking it can be effective. A useful side effect of 
this method is that it makes it possible for other users to reply, which can lessen your 
overall support load. This is most likely to occur in an environment like a university 
where large numbers of people (i.e., the students) tend to be enthusiastic generalists 
with a low per-hour cost and some amount of free time, or at least time available for 
procrastination. 

Office: a consulting office or physical help desk, where users or customers can walk in 
and (one hopes) get helped while they wait. This is also a great place for distributing 
printed documentation. 

Web form: the obvious method for the late 1990s, which could generate email, trouble 
tickets, or (potentially) even voice mail (but why would you bother?), or just about any¬ 
thing else. But make sure that the form is easily findable on (or from) your organiza¬ 
tion’s internal Web site. 

Hallway chat: Try to avoid this one, because you’ll never remember all the details, you’ll 
forget about it entirely, or something else will go wrong and get in the way of address¬ 
ing the original request. (The even more problematic versions of this include the 
barstool chat, the running out the door “by the way, I’ve got a question,” and the off¬ 
hand comment in the washroom.) I always make it a practice to say something like 
“Sure, we’ll get right on it, can you send some mail to 'request’ summarizing the situa¬ 
tion?” 

Regardless of the method used to make the request, the better and more complete the 
information you get with the request, the easier it will be to solve the problem; and the 
better you are at solving problems, the more reliable your systems and services will be. 
Consider the use of some form of problem-reporting form or checklist to help improve 
the quality of your initial data collection. 
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Tracking and Resolution 

Once you receive a request, you must use some method to keep track of it to make sure 
that it doesn’t get forgotten. Regardless of how simple or how sophisticated your track¬ 
ing system is, it will probably involve the following six activities: 

■ Recording - the initial information received from the user 

■ Delegation - assigning the task to someone 

■ Tracking and note-taking - during the investigation of the problem 

■ Resolution - an indication that the problem or request has been fixed or addressed 

■ Reporting - notification to the user of resolution and ongoing efforts if still in 
progress 

■ Archiving - a work record and (buzzword alert) “knowledge base” for future reference 

We often don’t analyze the process in such depth, but even those little pink telephone 
message slips can be used as the mechanism to implement those six activities. You 
might, however, appreciate the added features offered by even the most rudimentary 
automated tracking systems. 

The key benefits of a problem- and request-tracking system include: 

■ Reassurance for the user or customer by the assignment of a ticket number, or by 
query and status tools and messages if they’re available. It provides an indication that 
you take user/customer response seriously. 

■ An ongoing work record, which is useful for keeping track of changes, for standard 
answers and fixes, and as a customer service log. It’s far easier to convince manage¬ 
ment that you need more people if you have reliable statistics to back up your claim 
of being overworked. 

■ Provision of a to-do list and a mechanism to ensure that nothing gets lost. But make 
sure that you review the list - it won’t help if you just record the request and then 
forget all about it. 

■ Enabling outstanding requests to be escalated for more focused attention (even if it’s 
just your boss reviewing the two-month-old requests and asking you pointed ques¬ 
tions). 

■ A conversation/interaction history, which makes it far easier to pass a task on to 
someone else and get them up to speed. 

■ Aid in assuring a measure of consistency in the way you respond to requests, which 
will lead to more effective and efficient support. 

■ A training tool for staff. If they can review what others have done before them, they 
can do better themselves. 

All of these improve your reliability. 

Your Response 

I mentioned the need to report and reply to the requester, but it’s worth repeating. The 
purpose of all this effort is to reply to the requester, providing advice, a fixed bug, a 
plan, a report, or even a statement or apology that you’re unable to help (because of 
resource constraints, different areas of responsibility, and sometimes impossible prob¬ 
lems). Your response should be timely, complete, and correct, and it’s almost always 


A modest collection of links to problem- 
and request-tracking systems is at 
<http://www.net/-jsellens/tracking/>. 



February 1999 ;login: 


35 


SAGE NEWS & FEATURES 





One of the best ways to 
make yourself (or your 
group) more effective is to 
help your users and 
customers to help 
themselves whenever 
possible. 


worthwhile to provide interim status reports if it will take a nontrivial amount of time 
to respond to a request. Consider the method you use to deliver your response; choose 
among email, paper, phone, or face to face, depending on the problem, the response, 
and the person who made the initial request. 

Education, Training and Publications 

One of the best ways to make yourself (or your group) more effective is to help your 
users and customers to help themselves whenever possible. A one-time investment in 
training materials (with a small amount of ongoing maintenance and revision) can pro¬ 
vide a lasting positive effect on the effectiveness of all involved (service provider and 
service consumer). To tie this more clearly to reliability, a more effective user communi¬ 
ty will make fewer errors and will lead to a more reliable organization, and a system 
administrator who has fewer user requests (because of a better user education, docu¬ 
mentation, and training program) will be better able to deal with the systems them¬ 
selves, the long-term planning, and the problem avoidance that all contribute to higher 
reliability. 

Consider some of the following mechanisms for getting the word out: 

Email or Newsgroup Postings 

These are probably most appropriate for notices, brief announcements, etc., and usually 
are not very effective as an educational tool. One exception I have seen (as mentioned 
above) is the use of a local newsgroup for posting and resolving problems, which often 
provides a good learning environment for the innocent bystanders reading the group 
for entertainment. 

UNIX Man Pages and Other Traditional Help Systems 

As more computing activity happens in a GUI workstation environment, the traditional 
text-based help systems and man pages are losing some relevance for end users. But 
these methods are still very important for system administration and other behind-the- 
scenes activities, and also for people working in a command-line environment. And it is 
of course possible to put a Web front end on traditional help text. 

Web Pages 

Its probably a fair bet that the vast majority of internal systems documentation is being 
put on the Web now. For client-server, general process documentation, help desk, desk¬ 
top support, and the like, theres probably no better alternative 

Local Guides 

Local booklets and user guides can be very useful, as long as you have a reasonably sized 
topic area that doesn’t need constant updating. An example might be a user and securi¬ 
ty policies document. While you would almost certainly want to make an online version 
available, there’s still a lot of value in printed materials. I’ll mention the SAGE “Short 
Topics in System Administration” series as a successful and effective example of local 
community guides (as long as you’re willing to agree that SAGE is primarily a commu¬ 
nity). 

One-page Guides 

A number of universities and large organizations have developed very effective single¬ 
page topic guides, intended to be handed out from the help desk or consulting office. 
These are typically intended to be almost complete references to every (local) thing you 
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need to know, on topics such as setting up PPP, introduction to email, and how to print. 
When someone comes in to ask a question, you can provide the answer and also send 
them away with prewritten instructions. 

Classes and Tutorials 


Any training or information 
mechanism will take time 


Of course, sometimes there is no good substitute for good old-fashioned face-to-face 
learning, in a classroom or workshop setting. These are probably more effective for larg¬ 
er, more involved topics, or for areas in which it’s important to get a fast start. A com¬ 
mon example is when major new applications are put into place (such as a new pur¬ 
chasing system), and people have to be up and running almost immediately. 

Any training or information mechanism will take time to put together and get going. 
Resist the urge to put it off for another day. If you cant find the time yourself, hire a 
contractor or a third-party firm, or buy prepackaged course materials. The time you 
save may be your own. 


to put together and get 
going. Resist the urge to 
put it off for another day. 
If you can’t find the time 
yourself, hire a contractor 



Next Time 

In the next, and most likely last, “On Reliability” article, I plan to cover certain aspects 
of security and review how your security policies and practices affect the reliability of 
your systems. As always, if I’ve left something out, or I’ve forgotten your favorite topic, 
Td enjoy hearing from you. 

References 
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In the previous issue of ;login:, we began to explore the topic of how the Bourne 
shell and make can be used to maintain and generate collections of Web pages. 
In Part 1, we covered how the shell - or practically any scripting or macro lan¬ 
guage - can satisfy the objectives of consistency and simplification: variables 
and functions can be defined in a master script and in individual “source” files 
to hold values and consolidate HTML coding. In fact, the full power of the shell 
can be exploited to simplify many tasks. 

In this second installment, weTl discuss how make fits into this picture. This will not be 
a tutorial on writing makefiles, though relevant makefile features will be presented. 

Using make for Automation and Mutability 

We can exploit the UNIX make utility to great advantage, both to automate the genera¬ 
tion of the Web pages from their source files and the master script, and to help in the 
production of alternative builds of the Web pages. For those unfamiliar with make, let 
me just say that it will update designated “target” files (for example, recreate a compiled 
binary executable) based on any changes that have been made to any source files on 
which they depend, and utilizing implicit or explicit production rules. See man make for 
the details. 

Just as various approaches might be taken in the shell-scripting end of this process, var¬ 
ious approaches can be taken when constructing a makefile (a specialized sort of script 
for make, usually in a file named “makefile” or “Makefile”). I’ve chosen to construct my 
makefiles in a way that makes adding a new Web page easy: I only have to add the base- 
name of the new source file to one line in the makefile! 

Here are the components that are necessary, or that at least make things a lot easier. And 
though there are many platform-specific capabilities and features of make, the ones used 
here should be fairly portable. Please note that I’m no make expert, and so there might 
be more elegant ways to implement some of this. 

The first section of code, below, sets up macros to identify our (fictitious) Web pages. 
The first line is just a list of the basenames of the files, which should be the same for 
both those that contain the HTML source and those that contain the final HTML code. 
The second and third lines use a substring replacement macro feature to make lists of 
both the HTML source files (with a .hsrc suffix), and the HTML files (with a .html 
suffix). If we need to add a new Web page, we just add its basename to that first line. 
That’s it, the exception being if a page has any special processing needs (see below). 

NAMES = index parti part2 part3 notes appendix 
HSRCS = $(NAMES:=.hsrc) 

HTMLS = $(NAMES:=.html) 

The next section defines an implicit (default) build rule. The first two lines establish an 
implicit dependency of any .html file on an associated .hsrc file. The second two lines 
define the default command to execute to generate a .html file from a .hsrc file. This 
is all in cryptic makefile-ese , which we won’t delve into here. 

.SUFFIXES: 

.SUFFIXES: .html .hsrc $(SUFFIXES) 

. hsrc. html: 

./gen_html.sh $* 
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The next section defines some targets. The first target docs just says to make sure that 
all of the .html files are up to date. This might be the first (default) target in the 
makefile, and can be invoked with make docs or just make. The second one says that 
all of the .html files are dependent on the master script in addition to their default 
dependencies (their corresponding .hsrc files), and that they will all be rebuilt if it 
changes. The third line establishes a dependency of the file basenames on the .html 
files, and is the one that lets us specify a basename as a target; for example, make index 
instead of make index.html. The build rule for index is actually empty, but this still 
results in its dependencies and their dependencies (and so on) to be checked and 
rebuilt if necessary. 

docs: $(HTMLS) 

$(HTMLS): gen_html.sh 

$(NAMES): $$@.html 

With these definitions, rules, and dependencies in the makefile, we can now type make 
in the directory where these files reside, and any of the HTML pages that are out of 
date will be expeditiously updated from their respective source files, the master script, 
and any supplemental scripts and files. Please note that this is by no means the com¬ 
plete makefile. I’ve only included the parts here that warranted discussion. See the man 
page or other reference materials for more details on make and makefile fabrication. See 
the end of this article for URLs for a couple of sample makefiles. 


I’ve chosen to construct 
my makefiles in a way that 
makes adding a new Web 
page easy: I only have to 
add the basename of the 
new source file to one line 
in the makefile! 


Additional Dependencies 

In the previous article, I mentioned the example of the alumni address databases. An 
additional consideration in this and similar situations is that when we update one of 
these databases, we also need to rebuild the .html files that contain the data, that is, 
that depend on them. Its also a good idea to update these pages if the database conver¬ 
sion script is modified. We can accomplish all of this by providing some additional 
explicit dependency lines, such as: 

email-addrs.hsrc: email-addrs.db 
touch $@ 

email-addrs.html: cvt_addrs.sh 

The first two lines say that when the database changes, the .hsrc file should be marked 
as modified. The underlying reason why we want to do this is because we want the 
“Last update:” line in the Web page to reflect this change, and that date happens to be 
based on the modification time of the .hsrc file. This also indirectly establishes a 
dependency for the .html file. The third line says that to the current dependencies of 
the .html file, the conversion script should be added. In this arrangement, a change to 
the conversion script will not alter the “Last update:” date. That’s just a design decision. 
If we want it otherwise, we could just add cvt_addrs. sh to the first line, and omit the 
third line. 


Alternative Builds 

Now let’s say that the versions of the Web pages in our current directory are intended 
as prototypes, and that the real pages will contain different hypertext links and will live 
in a different location - possibly on a totally different network and HTTP server. We 
can adjust our HTML source and makefile somewhat to accommodate both versions, 
such that we can generate either on demand. 

The way that I’ve done this (and, again, there may be a better way) is to set up the 
makefile so that we can have it hard-link all of the relevant files (including itself) to a 
subdirectory, and then have it call itself with an argument (a macro definition) that 
causes an additional argument to be passed to gen_html. sh. This, then, tells 
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gen_html. sh to use a different BASE HREF for all of the Web pages it generates, and 
thats about all it takes. 

The additional lines in the makefile might look like this: 

AUX_FILES = Makefile gen_html.sh cvt_addrs.sh 

DB_FILES = email-addrs.db web-addrs.db 

DIST_FILES = $ (AUX_FILES) $(DB_FILES) $(TMPLS) 

DIST_DIR = production-dir 

## clear the distribution and install all links 
install-dist: 

@ echo "Erasing... " ; \ 

cd $ (DIST_DIR) ScSc rm -f $ (DIST_FILES) $ ( HTMLS ) 

@ echo " Linking..." ; \ 
cd $ (DIST_DIR) && \ 

for F in $ (DIST_FILES) ; do In ../$$F ; done 
## update dist html pages; assumes that links are up to date; 
make-dist : 

cd $ (DIST_DIR) && $ (MAKE) docs DIST= " -dist - 
## completely wipe, install, and make the dist html pages; 
all-dist: install-dist make-dist 

And in this makefile, the rule for building the .html file will instead look like this: 

.hsrc.html: 

./gen_html.sh $(DIST) $* 

When we make the distribution version with “make all-dist”, make calls itself with 
‘DIST="-disf 7 which causes gen_html. sh to be called with “-dist”. All gen_html. sh 
has to do is catch this option, and take some appropriate actions such as setting some 
variables, particularly the one that sets the BASE HREF. Now all that we need to do is 
copy the newly generated set of .html files from the subdirectory to their new home! 

So, maybe I’ve gone a bit more in the tutorial direction with makefiles than I had 
intended, but I feel that these examples are beneficial in building the case on the funda¬ 
mental role that the make utility can play in the design and upkeep of a collection of 
Web pages. I hope I’ve convinced you. 

Wrappin’ Up 

There’re a lot of free programs and packages out there on the Web (speak of the devil) 
to help you to develop Web pages, not to mention the plethora of commercial packages. 
Some are specialized macro packages, some are WYSIWYG editors, others are server- 
side dynamic page generators. (I haven’t used any of them; let me know if you have any 
recommendations.) If you care to explore, you might try searching on “HTML editors” 
or “HTML authoring” (or substitute “Web” for “HTML”). A good starting point for 
surfing might be <http://www.w3.org/Tools/>, though unfortunately this otherwise excellent 
page is no longer being updated. 

Of course, for us do-it-yourselfers, a shell and make have a certain appeal. I’ll provide 
some sample scripts and makefiles on the Toolman home page and FTP site; you can 
download them for use as starting points, if you’re inclined to experiment with these 
ideas. 

Happy scripting! 

URLS: 

<http://www.cs.duke.edu/-des/toolman.html> 

<ftp://ftp.cs.duke.edu/pub/des/scripts/gen_html/INDEX.html> 
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the first ISP 


It's not my fault. But sometimes I think it might be. My name is Spike. You've 
probably never heard of me, but in 1989 I did something that had an impact 
on the Internet today. I started the first ISP. 

Actually I didn't, Barry Shein did, I just actually made it work. In 1989 Barry started a 
small UNIX consulting company call Software Tool & Die. There wasn't a lot of con¬ 
sulting that summer, but there were a lot of calls for people who wanted to know where 
they could find access to email and USENET News. 

You might not know what USENET News is either, but it was the reason people got 
online before someone thought up the Web. Lots of discussion groups with lots of peo¬ 
ple posting messages. Not in realtime, mind you. But it was like, as someone more 
clever than I once said, the world's biggest cocktail party. 

STD (yes, I know) had really good news. Barry is, was, and always will be a USENET 
junkie. Word must have gotten around 'cause people who d lost their access at school 
(pretty much the only way to get it back then) came nosing around looking for access. 
Barry had an idea: let's sell it to them. 

It wasn’t really a new idea. There was The Well in San Francisco, Portal might have been 
up and running by then, and by that time countless BBSs (which you may have never 
been a part of either). But nothing east of the Mississippi that we knew of. 

So off we went to the BitBucket to buy six 2400bps modems (with MNP 5 and maybe 
Retsyn). Then came a number of sleepless nights while I wrote account-creation soft¬ 
ware, installed all the software our UNIX-hungry future customers might want, made 
modem cables (really) to connect up those modems to a Sun (like they made Toy Story 
with only much bigger and much slower and, well, only one), and drank a lot of Coke. 
Jim Frost wrote billing software, drank Mountain Dew, and ate Cheeze-its. 

On a fine Boston (OK, Brookline) day early in November 1989, our first customer 
logged on and The World was born. 

Those were the days. I was the sysadmin, the tech support guy, and, when Jim moved 
on to greener pastures (this is before there was any money in the Internet), the system 
programmer too. I drank a lot of Coke and ate a lot of ramen noodles. 

Now, back then if you weren’t an institute of higher learning or a defense contractor, 
you couldn't be on the “Internet.” You got your email and news using UUCP, old-style. 
With UUCP you called other computers and exchanged mail and news with them, and 
they called other computers and passed it on and so on and so on. Clunky, but it 
worked. Someone once said, Never underestimate the bandwidth of a station wagon 
full of 1/4" tapes (think “minivan full of Zip disks”), and the same can be said for lots 
of 2400 and later 9600bps (sort of) modems. 

But all things change. Sometime in 1990 the National Science Foundation decided 
they'd like less involvement in running the Internet (you didn’t even know the NSF ran 
it, did you?). So they began what is popularly called, in Socialist countries, privatizing 
the Internet. One company to take advantage of this was UUNET, the king of the 
above-mentioned UUCP. They formed a new company called Alternet to provide 
Internet service. At this point, gentle reader, you'll be able to guess who Alternet turned 
to when they needed space in Boston (OK, Brookline) to install equipment. 

Naturally we got connected as part of the deal. 
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Everything good was 
always somewhere else 
(and there wasn't even a 
Web yet). 


But it wasn’t that simple. In 1990 the NSF still ran the Internet backbone (which 
despite what you hear every day doesn’t exist anymore) and you still had to be an insti¬ 
tute of higher learning or a defense contractor to be connected. We could get to the 
parts of the Internet that were connected to Alternet and a few other networks that 
were privately interconnected with them (they call it peering in the modern era), but 
not the bulk of it. Frankly, this sucked. Everything good was always somewhere else 
(and there wasn’t even a Web yet). Even worse, try explaining this paragraph on tech 
support calls all day. People always want what you ain’t got. 

If you ve been around for a while and have read USENET, and in that unlikely event 
read something other than porn (yep, USENET has more porn that you can shake a 
stick at), you ve probably come across Barry Shein. And if you’ve come across Barry 
you know not to argue with him or at least that you’d better have your facts straight 
and be damn sure of them. The man has forgotten more than I’ll ever know, has a bet¬ 
ter command of the English language than Danielle Steele, drinks way way too much 
coffee, and somehow knows how everything comes down to the Battle of Hastings 
(1066 A.D.). Much of my skill as a business person comes from all those years of trying 
to explain to Barry why my idea was better. 

Anyway, in late 1992 Barry turned his awesome powers on the NFS. And eventually 
they relented. The NSF allowed companies to sell dialup access to the Internet with 
some lame disclaimer about how those customers would probably be working with an 
institute of higher learning or defense contractor. 

So that’s how on August 13, 1992,1 was running the very first official ISP. Oh, sure it 
was only a day or two before the other guys got their permission. Hell, by the next year 
the rules had been pretty much thrown out all together. 

But there you have it, the first ISP. 

Nine years have passed. I now run my own ISP in lovely (whiter whites, brighter colors) 
Boulder, Colorado. The World is still there and so is Barry, though it’s now a Silicon 
Graphics (like they used for Jurassic Park). When they write the history books of the 
Internet, my name won’t come up. Maybe The World will, maybe it won’t. If it hadn’t 
been me it would have been someone. But I was there and / did the deed and somehow, 
just maybe, as I filter the spam out of my inbox and see the Web address on the side of 
trucks and cans of soda, it may be partially my fault. 
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java performance 

Memory Fragmentation 

Suppose that you are developing an application, one that will make use of very 
large arrays of objects. If you’ve studied the area of memory allocation at all, an 
issue that may be familiar is fragmentation: Adequate memory is available to 
an application, but the memory is broken into small chunks, none of them big 
enough to satisfy a given memory request. For example, the application may 
request 1000K of contiguous memory to represent some data structure, and 
the request will fail because only four chunks of 250K each are available. 

This problem has an obvious solution, which is to represent the data structure in pieces 
and map indices into the structure into the appropriate piece. Such a technique will 
work with any programming language, but languages like C++ and Java make it easier, 
because the details of representation and mapping can be encapsulated within a class. 

To see how this works, consider a Java class to represent very large or sparse arrays of 
objects: 

import java.util.*; 

public class SparseArrayList extends AbstractList { 

// default page size 

private static final int DEFAULT_PAGE_SIZE = 1024; 

// actual page size that is set 
private int pagesz; 

// pages 

private Object pages[][] = new Object[0][]; 

// default constructor 
public SparseArrayList() 

{ 

this(DEFAULT_PAGE_SIZE); 

} 

// constructor specifying page size 
public SparseArrayList(int sz) 

{ 

if (sz < 1) 

throw new IllegalArgumentException(); 
pagesz = sz; 

} 

// number of slots currently allocated 
public int size() 

{ 

return pages.length * pagesz; 

} 

// set an array slot to a given value and return the old value 
public Object set(int index, Object val) 

{ 

if (index < 0) 

throw new IllegalArgumentException(); 
int p = index / pagesz; 

// if page array not big enough, expand 

if (p >= pages.length) { 

Object newpages[][] = new Object[p + 1][]; 

System.arraycopy(pages, 0, newpages, 0, pages.length); 
pages = newpages; 

} 
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// need to allocate a new page? 

if (pages[p] == null) 

pages [p] = new Object [pagesz]; 

Object old = pages[p][index % pagesz]; 
pages[p][index % pagesz] = val; 

return old; 

} 

// get the value of a given array slot, null if none 
public Object get(int index) 

{ 

if (index < 0) 

throw new IllegalArgumentException(); 
int p = index / pagesz; 

if (p >= pages.length II pages[p] == null) 
return null; 

return pages[p][index % pagesz]; 

} 

} 

Java 2 contains a set of classes known as a “collections framework,” and the 
SparseArrayList class is based on this framework. Collection and List are top- 
level interfaces (an interface is a specification of a set of methods that an implementing 
class must declare and implement) in the framework, and the abstract classes 
AbstractCollection and AbstractList partially implement the functionality of 
these interfaces. SparseArrayList need only define constructors and the size(), 
get () , and set () methods in order to fully implement the interfaces. 

SparseArrayList represents an array as a series of pages, each containing a default of 
1024 elements. An array index is split apart to find the page and offset within the page!. 
A page is allocated only if needed. An array of 1 million elements will use 977 pages (1 
million / 1024), so the array will be represented as a page array 977 long, together with 
a series of object arrays each 1024 long. Note that specifying very small page sizes may 
result in frequent reallocation and copying of the pages array, hurting performance. 
One solution to this problem would be to define a constructor that specifies the maxi¬ 
mum size of the array in advance, and another solution would be to grow the pages 
array by more than one page slot at a time. 

A simple program that exercises this class is: 
import java.util.*; 

public class testarrayl { 
public static void main(String args[]) 

{ 

Random rn = new Random () ; 

List sal = new SparseArrayList{); 

for (int i = 1; i <= 1000000; i++) { 

// generate a random number 
int r = rn.nextlnt(1000000) ; 

// set its array slot and then 
// retrieve and compare 

sal.set(r, new Integer(r)); 

if (((Integer)sal.get(r)).intValueO != r) 

System.err.println(r); 

} 

} 

} 
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There are obvious advantages to using a class to encapsulate a data structure. But what 
about the costs of doing so? Accessing an array through a method is slower than using 
primitive machine operations. One answer to the question is to come up with a simple 
benchmark: 

import java.util.*; 

public class testarray2 { 

public static void main(String args[]) 

{ 

List sal = new SparseArrayList() ; 

Object obj = new ObjectO; 

for (int i = 1; i <= 1000000; i+ + ) { 
sal.set(i, obj); 
obj = sal.get(i); 

} 

} 

} 

This program sets and then gets 1 million elements from a SparseArrayList structure. 
It requires around 1.35 seconds to run on a 300MHz Pentium, or 1.35 microseconds 
per iteration. For 8 million elements, the time increases to 1.7 microseconds per itera¬ 
tion, reflecting the extra time to reallocate the page table. 

One final point about SparseArrayList. Because it is designed to operate within the 
collections framework described above, an object of type SparseArrayList can be 
specified as a method argument wherever a Collection or List is called for, and stan¬ 
dard operations such as iterators are automatically available. For example, the following 
program creates a SparseArrayList and then sets the values of the first few elements. 
Then the element values are dumped out via an iterator. 


import java.util.*; 


public class testarray3 { 

public static void main(String args[]) 
{ 

List sal = new SparseArrayList(10); 


} 


sal.set(0, 
sal.set(1, 
sal.set(2, 
sal.set(3, 
sal.set(4, 
sal.set(5, 


new Integer(0)); 
new Integer(1)); 
new Integer(4)) ; 
new Integer(9)); 
new Integer(16)); 
new Integer(25)) ; 


Iterator iter = sal.iterator() ; 
while (iter.hasNext()) 

Systern.out.println(iter.next()); 


The output is: 

0 

1 

4 

9 

16 

25 

null 

null 

null 

null 

The iterator mechanism uses size () and get (), and thus the values for the whole page 
are reported. 
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intrusion-detection 

systems 

Guaranteeing the Safety of a Network 
Beyond Using a Firewall 

The increase in the number of links with the Internet has generated a propor¬ 
tional increase in the number of attacks brought against them. Implementing 
systems capable of detecting these attacks (possibly without false alarms, 
which in some cases is an impossible dream), and also giving the system 
administrator the opportunity to intervene immediately, is a specific area of net¬ 
working research: intrusion detection. This article discusses how and why intru¬ 
sion-detection systems (IDSs) should be used to record and report possible vio¬ 
lations of information systems linked to the Internet. 

IDSs can be considered an extension of network analyzers. Network analyzers are 
devices that monitor and analyze the network traffic in realtime, for the purpose of 
identifying any arbitrary violations of general policy established by the network manag¬ 
er. The differences among them concern the methods used for analyzing and reporting, 
and the type of traffic that can be identified. The task of an IDS is to record and report 
any violations of information systems (even in the form of attempts). They are not per¬ 
manent devices. In other words, they do not constitute a replacement for a firewall, a 
proxy, or similar devices. IDSs are installed in addition to firewalls and carry out a 
check at an internal level, on the customer side of the network. 

Generally speaking, IDSs are grouped into two families: the first performs mainly 
auditing functions, carrying out continuous calls to the hosts linked to the system. The 
second works independently, observing the traffic on the networks directly (and pas¬ 
sively), carrying out “usual” packet filtering. Some commercial products combine the 
two methods. 


Many IDSs feature automatic network-attack recognition and response-system type. 
The IDSs are installed and made to “run” throughout the network or on parts of it 
where there is a need to preserve critical information. 

Monitoring of data traffic generally takes place on the TCP/IP stream passing through 
Ethernet-based infrastructures. (Some manufacturers are working on supports for dif¬ 
ferent layouts.) Thus, the IDS combines the functions of a simple network analyzer 
with that of recognizing attacks. This is generally achieved by means of a signature 
check, rather like programs that detect viruses. The system contains a database of 
attacks, which functions as a basis of comparison with the traffic analyzed. When an 
attack is recognized, the IDS can stop the stream of data immediately, thereby prevent¬ 
ing the attack from causing damage to the network. 

An IDS that does its job properly must also enable recording of the date, source, target, 
and type of the attack, and of the activity undertaken. As for the source of the attack, 
the IDS must implement antispoofing functions. This turns out to be very important, 
primarily for preventing some DOS (denial-of-service) attacks. 

The basic principle of operation of IDSs is prompt intervention. In practice, once the 
attempt to intrude has been discovered, the system must warn the person in charge of 
corporate security, who must be able to implement the necessary countermeasures, 
even from a remote location. It is possible for the system itself to give a warning of the 
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incoming attack, for example by means of an SMS message (texts on GSM) or a warn¬ 
ing sent to a pager for the security administrator. 

As to required computing capacity (ascertainable in terms of the slowdown it causes in 
the performance of the whole network), an IDS must be able to facilitate maximum 
scanning speed and therefore present minimum requirements in terms of latency. 
Although this is guaranteed by most manufacturers, discussions among administrators 
on this issue are focused mainly on this aspect. 

From the point of view of development philosophy, the most widespread analysis 
model used by the IDSs currently being circulated is called CIDF (Common Intrusion 
Detection Framework). This model consists of splitting up the IDS into a series of 
components, called boxes, which range from the management of suspicious occur¬ 
rences, to the countermeasures adopted, to safe storage of the logs generated. These 
boxes, which are generally preceded by a letter indicating their function, are of funda¬ 
mental importance also for legal purposes; regulations concerning computer crimes 
may require the log files as proof of illegal access. 


The task of an IDS is to 
record and report any 
violations of information 
systems (even in the form 
of attempts). They are not 
permanent devices. 


Why Get an IDS? 

While most attacks come from outside, badly administered and controlled internal 
resources can be a source of large-scale problems. That is the main reason why, at 
times, installing a firewall is not sufficient to limit the damages. In addition, a firewall 
can crash or, worse still, it might have been improperly configured. In that case, it is 
important to have a rearguard product able to interact with the perimeter of the 
system. 

Computing Demands and Band Requirements 

It must be possible to control the central unit and management from dedicated 
machines. For this purpose, for example, RealSecure Engine requires a minimum 
200MHz Pentium, 32MB of RAM, at least 199MB of disk space for the log files and 
databases, 10MB for the software, and an Ethernet NIC operating in promiscuous 
mode. A remote management console, on the other hand, requires, for an IDS working 
on Intel systems, a 200MHz Pentium, 32MB of RAM, and 100 MB of disk space for 
each scanning engine it manages. 

Arguments about the Effectiveness of IDSs 

To what extent can an IDS influence the performance of a network? How effective can 
it really be? Management wonders most about such issues. Two well-known American 
security experts, Thomas T. Pracek and Timothy N. Newsham, have recently triggered a 
discussion about the true effectiveness of IDSs. According to these two experts, these 
types of device have many “weak points.” However, most of these Achilles* heels reside 
in the analysis models. 

In considering IDSs based on the signature-analysis method, also known as misuse 
detection, Pracek and Newsham have raised the objection that often the policies imple¬ 
mented are too stiff and are linked to mere signature matching; that is, in practice they 
are based excessively on passive monitoring. Errors made during configuration can lead 
to a burst of false alarms, particularly when specific services are used. To this end, the 
two experts, who state that they are more in favor of the active-proxy monitoring type 
of setup, feel that maximum granularity of the IDSs based on this method is required. 
As they have pointed out, highlighting the possible leaks in the analysis and operational 
model of an IDS does not label these devices as irreparably unsafe - not at all. Instead, 
it points to a series of improvements to make to products in order to make these sys¬ 
tems as safe as possible. 
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Some analysts compare 
IDSs with the first virus- 
detection programs. 


Conclusions 

The analysts feel that the current status of IDS technology can be compared to that of 
firewalls a few years ago. Some of them compare IDSs with the first virus-detection 
programs, with all the problems they used to have. Perhaps, they stress, the solutions 
available on the market are still “immature” It should be considered, however, that 
products such as those surveyed in the sidebar will have something authoritative to say 
within a short time, or might even become reference points. Apart from anything else, I 
feel that the optimization process of these systems will be accelerated to an extent that 
is directly proportional to the commitment the international academic community will 
put into studying the problems involved. 


A Survey of the Best 

CyberCop 

CyberCop, originally produced by Network General, is an IDS 
recently released by Network Associates <www.nai.com>, an organi¬ 
zation that is constantly at work acquiring technologies in the 
field of security. After their acquisition of Network Generals 
product, Nai made some changes to CyberCop. (It must be 
remembered that the auditing/scanning software now called 
CyberCop Scanner - also known as Ballista - is different from 
the totally software-based product we are discussing here.) 

Installing CyberCop does not require the network to be reconfig¬ 
ured or plug-ins to be added. Like other IDSs, CyberCop builds a 
layer of additional software which works by monitoring the ports 
and services enabled by the firewall. 

The first version of CyberCop, announced in 1997, consists of 
two elements, the management server and the sensors. The latter 
are positioned at strategic points on the network and communi¬ 
cate any suspicious events to the management server. These 
events are classed according to a set of 170 different attacks. 

If an attempt is made to access the network, the product, cur¬ 
rently called CyberCop Server, informs the security administrator 
in real time, providing a detailed report of the event. The design¬ 
ers feel that within a few minutes CyberCop can give the input 
the security manager needs to take the necessary steps to resolve 
the problem. Management of the configuration of CyberCop, as 
well as the receiving and transmitting of the intrusion detection 
reports, can take place from a remote location using an encrypt¬ 
ed link, which is activated only after recognition of the parties. 

Of course, all the traffic monitored is stored in log files which 
can be consulted at any time by the security manager, both in 
order to trace the attacks and in order to take subsequent legal 
action. Configuration and positioning of the sensors are simpli¬ 
fied by a preconfigured installation set, which makes operation 
easier and enables leaks to be limited. 


known IDS Products 

Bro 

Bro is a realtime IDS devised and developed by Vern Paxson and 
other experts at the Network Research Group of the Lawrence 
Berkeley National Laboratory. The source code of Bro is freely 
available, and the principle on which it is based is decidedly in an 
academic mold. With its spartan interface, indicating that greater 
attention has been paid to substance than to appearance, Bro 
bases its operational capacity on its scanning speed, realtime 
notification of violations, and a clear separation between the 
engine, the policy implemented, and the extensibility options. 

Bro is partitioned into two components: the “event engine,” 
which translates the traffic intercepted at kernel level into high- 
level events, and the “policy script interpreter,” which defines the 
policy implemented, always by means of specific instructions 
written in a proprietary language. In this way, administrators can 
use the granularity of this IDS to adapt the system to their own 
requirements. The services monitored on a priority basis by Bro 
are Finger, FTP, and Telnet. In addition, the Portmapper function 
of this solution makes it possible to check the activity of the sin¬ 
gle ports as well. 

So far we could say that there is nothing new. All network analyz¬ 
ers (or net sniffers, if you prefer), and therefore all IDSs (which 
can be considered extensions of them) are normally equipped 
with these features. The designers of Bro, however, state that dur¬ 
ing the analysis period they studied in depth the typology of 
both standard attacks and those that can be brought to bear on 
the screen in the narrowest sense, and that they were able to 
identify and describe attacks not referred to in the literature. 
Again, during the prebuilding phase the designers acquired sub¬ 
stantial experience with systems based on offline analysis of tcp- 
dump attempts. All this has given rise to a melting pot of refer¬ 
ence information for subsequent implementation of the modules 
of this IDS. 
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One of the main objectives of Bro is to ensure traffic speed. In 
order to do this, Bro monitors DMZ links. These are usually 
FDDI, so that the monitor must be able to inspect the traffic, 
which is very bulky in itself, at speeds in excess of 100 Mbps. 

Bro’s separation of the engine from the rest of the modules, 
including the script policy interpreter, is essential to streamline 
the monitoring operations as much as possible (which means no 
degradation of network performance) and to distinguish the data 
on the basis of the services to which they belong. All this has 
been implemented in order to give Bro maximum flexibility. 
(Flexibility is more or less the reason why the manufacturers of 
virus-detection programs are revolutionizing the way they are 
developed. Attacks are becoming increasingly numerous and 
diversified, and they depend more than ever before on flaws in 
individual operating systems and their layouts, so they are 
increasingly well-targeted and unforeseeable. In this context, 
modularity and extensibility are strategic and can only be 
achieved if the architecture used is as open as possible.) 

More information about Bro may be found at <http://nrg.ee.lbl.gov/ 
nrg/papers.html>. 

ISS RealSecure 

ISS RealSecure for Windows NT by Internet Security Systems 
<www.iss.net> is one of the best known and best-selling IDSs on 
the market. (Indeed, ISS and Checkpoint have joined in partner¬ 
ship to bundle Firewall-1 and RealSecure together.) The basic 
operating principle is common to the other IDSs: The traffic 
passing through is monitored, and the activities are compared 
with the pattern with which it is outfitted. In the event that they 
match up, an alert is activated and possible automatic counter¬ 
measures are implemented. 

Suspicious activities, documented with information concerning 
the chronology of the attack, its source, and destination - plus 
other data to be selected - can be managed extremely dynamical¬ 
ly. Monitoring of the traffic consists above all of packet filtering. 

It is possible to configure RealSecure to check traffic in all its 
meanings: TCP, UDP, ICMP, source and destination ports, etc. It 
is also possible to check the traffic on the basis of the services 
used because the pattern of the attacks follows this schematic dis¬ 
tribution. 

The designers of ISS used this philosophy: Starting from the 
assumption that most attacks come from inside, the administra¬ 
tor needs a product able to check all the traffic (not only the traf¬ 
fic permitted by the perimeter security system). In addition, a 


check of the activity permitted by the firewall is also indispens¬ 
able, since even an authorized user can “penetrate” a system. 

The security policy set up by RealSecure therefore has the objec¬ 
tive of checking and identifying beforehand: 

■ who can access the system and who cannot 

■ which protocols and/or services are permitted 

■ which new hosts are added to the network and what rights 
they have to “dialogue” with the rest of the infrastructure. 

Starting from these assumptions, a series of features in Real 
Secure are aimed at making the work of the administrator as easy 
as possible and the system as flexible as possible. 

NID 

NID 2.x is an intrusion-detection suite available freely on the 
Web <http7/ciac.llnl.gov/ctsc/nid/> for various operating systems, 
including LINUX, but its use is limited, for the moment, to gov¬ 
ernment organizations. 

NID works in a manner similar to Bro. It can monitor speeds 
and layouts, including FDDI and, of course, all IP traffic. NID 
has these features: 

■ The software is installed on a dedicated machine. 

■ A security domain is formed from the management console. In 
turn, this includes a series of hosts at the discretion of the 
operator. 

■ NID starts to audit the network traffic using three fundamen¬ 
tal methods. 

■ Attack-signature recognition 

■ Vulnerability risk model, i.e., general safety parameters to be 
observed 

■ Anomaly detection, i.e., recognition of abnormal behavior 
inside the network and immediate notification of the system 
administrator 

In NID, too, the analysis model and its operational expression are 
of the mainly “passive” type; traffic is audited and a consequent 
comparison with the attack pattern at disposal is made. If a 
match is found, an alarm is sent to the security administrator. 

The software permits sessions of specific UNIX tasks, such as 
cron, to be run. 
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speedier squid: 

A Case Study of an Internet Server 
Performance Problem 

It’s not always easy to discover what limits the performance of an Internet serv¬ 
er application. Is the network overloaded? Is a Web page too complicated? Is 
the operating system inefficient? Is the application doing something the wrong 
way? 

People make guesses about why a server runs slowly, and their intuition is often very 
wrong. Internet servers, such as Web servers and proxies, are a particularly tricky 
domain for intuition, because their performance depends on so many different aspects 
of computer system design. We may not even have a good feel for how fast a particular 
server ought to be, let alone why it runs slower than we would like. 

This article is a case study of one such performance problem and what I did to solve it. 
Of course, I’ll discuss the solution, but my main goal is to show how one can combine 
an advanced performance-measurement tool and some understanding of computer 
performance to rapidly discover and solve a performance bottleneck. 

What Is Squid? 

Squid[7] is a Web (HTTP) proxy server. It was based on software from the earlier 
Harvest research project[4] and is being developed and maintained by a community of 
Internet users, under the supervision of the NSF-funded National Laboratory for 
Applied Network Research. The Squid source code is freely available and runs on a 
wide variety of UNIX-like systems, making it a convenient platform for research; as a 
result, Squid is one of the most widely used proxy-server applications, even in nonre¬ 
search environments. 

The primary goal of the Squid project is to provide a Web caching service. Squid 
servers are typically used as proxy caches, storing previously retrieved Web objects for 
later reuse. However, Squid can also be used as a noncaching proxy, typically as part of 
a security “firewall” between an organization and the Internet. For example, the Palo 
Alto firewall between the Internet and Digital Equipment Corporations internal net¬ 
works has used Squid as both a caching proxy and a noncaching proxy. (Although 
Digital has merged with Compaq, most or all of the users of this proxy are in the 
“dec.com” domain.) My experiments were performed at this firewall. 

When Squid is used as a caching proxy, its performance is greatly complicated by its use 
of disk I/O for storing, finding, and retrieving cache entries. In some contexts, such as 
the Digital Palo Alto firewall, where external network connectivity is reasonably fast, 
the disk I/O overhead of caching turns out to be much higher than the latency 
improvement from cache hits (which could be the subject of a future case study!). 

Thus, we typically run the Squid server in its noncaching mode. 

A noncaching proxy makes little use of the filesystem, but it still faces performance bot¬ 
tlenecks. As the HTTP request rate increases, so does CPU utilization. Since our proxy 
often handles 2.5 million requests on a busy day, with peak rates much higher than the 
average, it can't afford to spend much CPU time per request. More important, we want 
it to scale well: The CPU time per request should not increase significantly as the 
request rate increases. For example, if the cost per request is proportional to the request 
rate, then the system’s cost will increase in proportion to the square of the request rate 
(called 0(N 2 ) behavior). 

The use of Squid as a noncaching proxy stresses many aspects of the system, including 
those that also affect Squid caching-proxy performance. Thus, although this article 
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concentrates on noncaching performance, the results are still applicable to a caching 
Squid system. 

For these tests, I used Squid version 1.1.20. Later versions have been released, but this is 
relatively close to the current “stable” version of the software. 

Lab Tests versus Live Tests 

Many Internet performance tests are conducted in laboratory environments. For exam¬ 
ple, the SPECweb96 benchmark, used for measuring Web server performance, assumes 
that the clients are connected via a LAN. This assumption may be necessary for a feasi¬ 
ble benchmark test (because it is expensive and difficult to perform repeatable mea¬ 
surements at high request rates in the real Internet), but it can seriously skew the 
results. 

In lab tests of Web servers, connections typically last for a few tens of milliseconds. 
However, HTTP connections in the real Internet often take hundreds of milliseconds, 
or even several seconds, because of much longer round-trip times, higher packet loss 
rates, and lower link bandwidths. 

Consider a server handling 100 requests/sec. If the mean request duration is, say, 25 
msec., then on average the server will be handling 100 x 0.025 = 2.5 connections at 
once. But if the mean request duration is 2 sec. (not atypical for the real Internet), then 
the same request rate means that the server will be handling, on average, 200 connec¬ 
tions at once. If the server application uses algorithms that scale badly with the number 
of connections, or if it uses operating system features that scale badly, a request load 
that is easily handled in the lab may overwhelm the system in live use. 

The measurements in this article were made on our live proxy system, in order to 
reflect real-world performance issues accurately. This makes it harder to obtain repeat- 
able results and so requires some attention to experimental procedures. For example, 
trials must last long enough to avoid short-term fluctuations in load. But since the 
request load varies by a factor of ten between “prime time” (usually late morning on 
weekdays) and the middle of the night, its also important to decide whether one wants 
to measure performance over a wide range of offered loads, or during a period of rela¬ 
tively constant load. I used one-hour prime-time trials when I wanted to measure peak¬ 
load performance, and full-day trials when I wanted to measure performance over a 
wide range of loads. 

Tools in action 

Performance measurements are only as good as the tools one uses to obtain them. I’ll 
briefly explain the tools that I used. 

Measuring Load and CPU Utilization 

Since my primary concern was how much CPU time the Squid system uses and how 
this scales with request rate, I needed to measure CPU utilization over short periods of 
time and then correlate these measurements with the request rate. 

Most Web servers, including Squid, can record per-request information in a log. We 
already had software to analyze the Squid request logs (offline, so as not to add load to 
the server) and compute request rates over various intervals. I chose a 15-minute inter¬ 
val, which yields 96 samples over a 24-hour period. 

Our servers run shell scripts every 15 minutes to log various system statistics (including 
output from netstat, ps, vmstat, etc.). Although vmstat does display CPU utilization (by 
fraction spent in user mode, kernel mode, and idle time), in order to get 15-minute 
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The CPU-utilization 
measurements show how 
much CPU time is being 
spent at various request 
rates, but they don’t show 
where it is being spent. 

... To localize bottlenecks 
more precisely, one needs 
a procedure profile. 


totals for these fractions it proved simpler to write a short program that provides the 
total number of clock ticks spent in each mode since the system was booted. A postpro¬ 
cessing script computes the differences between subsequent samples, avoiding any 
errors introduced by clock skew in the sampling process. 

With measurements for both the request load and the CPU utilizations over identical 
15-minute sampling intervals, I could then plot the relationship between request rate 
and CPU utilization. For example, see Figure 4.1 could also run linear regressions to 
quantify the relationship. For example, see Table 2. 

DCPI for Procedure Profiling 

The CPU-utilization measurements show how much CPU time is being spent at vari¬ 
ous request rates, but they don’t show where it is being spent (aside from the coarse 
breakdown of kernel-mode time versus user-mode time). To localize bottlenecks more 
precisely, one needs a procedure profile. Profiling software has been around for many 
years, going back to early versions of UNIX[6]. However, most profiling tools have sev¬ 
eral drawbacks: 

■ They cover only one application at a time (and don’t include kernel functions). 

■ They count instructions executed, not cycles spent during execution. 

■ They require recompilation or post-processing of application binaries. 

■ They may add significant execution overhead. 

A system called DCPI[1, 5] (for “Digital Continuous Profiling Infrastructure”) has 
none of these drawbacks. DCPI, which is available for Compaq’s Digital UNIX and 
Windows NT on Alpha processors, consists of a dynamically loaded kernel device dri¬ 
ver and a set of other programs. It is capable of profiling the entire system, including 
the kernel, shared libraries, and all applications, while adding an overhead of only 1% 
to 3%. DCPI counts cycles, not instructions, so it accounts for the cost of delays such as 
cache misses and TLB misses. 

This makes DCPI an effective tool for profiling an Internet server, since it quickly and 
easily reveals the functions that account for CPU consumption, no matter where they 
lie. Table 1 is a DCPI profile of the unmodified Squid 1.1.20 software running on a 
modified version of Digital UNIX V4.0B, on a 500MHz AlphaStation 500 system 
(21164A processor, SPECInt95 = 15.0) with 512MB of RAM. The table includes all pro¬ 
cedures that account for at least 0.5% of the non-idle CPU time. The kernel modifica¬ 
tions, described in another paper[3], improved the performance of the kernel’s 
select () system call, although Table 1 shows that the various kernel procedures that 
implement select () still account for about 25% of the non-idle CPU time. 

However, the single most expensive procedure, in this profile, is the comm_select () 
procedure in the Squid program. In fact, no other nonlibrary procedure in Squid 
accounts for more than 0.69% of the nonidle CPU time, and all of the other Squid pro¬ 
cedures together account for less than 5% of the non-idle CPU time. This suggests that 
comm_select () is the best target for further analysis. 

DCPI for detailed analysis 

Suppose we want to compute the number of CPU cycles required to execute a particu¬ 
lar piece of code. We could start by counting instructions. However, most high-perfor¬ 
mance CPUs are able to issue multiple instructions in a single cycle, although resource 
limits and operation latencies usually prevent the CPU from filling all issue slots on 
every instruction. 
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More important, modern CPUs experience “dynamic stalls,” where 
they must stop issuing instructions for a short period. For example, if 
the CPU encounters a conditional branch instruction and has incor¬ 
rectly guessed whether the branch will be taken (a “branch mispre¬ 
dict”), its instruction pipeline must be refilled before it can continue 
at full speed. 

The lengthiest dynamic stalls usually occur when the memory system 
is unable to deliver data or instructions fast enough. While caches can 
help, most real applications don’t fit in on-chip caches. Main memory 
latencies are often measured in hundreds of nanoseconds, while fast 
CPUs have cycle times of 2 nanoseconds or less, so one cache miss can 
cost more than several hundred instructions. This is especially prob¬ 
lematic for operating-system code and similar programs, such as 
Squid, that to do relatively little arithmetic and that manage complex 
data structures. 

DCPI’s procedure-level profile can include the number of data-cache 
(D-cache) misses encountered by a procedure (not shown in Table 1). 

DCPI found that comm_select () encountered far more D-cache 
misses than any other procedure. Perhaps something in 
comm_select () is causing lots of dynamic stalls? 

DCPI includes a tool called dcpicalc that uses the information in a 
profile, together with an accurate model of the CPU, to deduce which 
instructions in a procedure are stalling, and why. The full dcpicalc 
output for comm_select () is too lengthy to include, so I will show 
some short excerpts. 

A common measure of CPU efficiency for a particular pieces of code 
is the average “cycles per instruction” (CPI). For example, a dual-issue machine with no 
stalls can, at best, execute at 0.5 CPI. Dcpicalc found, by analyzing the comm_select () 
profile against a model of the Alpha 21164A CPU, that the best-case execution could 
have averaged 0.71 CPI, but the actual execution averaged 2.69 CPI. 

Dcpicalc then analyzed each profiled instruction to deduce how often it stalled, and 
why. This analysis is complex and based somewhat on heuristics, but generally gives 
plausible results. It found that 71.4% of the cycles spent in comm_select () went to D- 
cache misses, and only 3.7% went to other dynamic stalls. What is causing those D- 
cache misses? 

Figure 1 shows excerpts from dcpicalc’s annotation of the worst basic block in the 
comm_select ( ) profile. (A basic block is a set of instructions always 
executed as a group.) Dcpicalc starts by saying that this block could 
have executed at 0.67 CPI in the absence of any dynamic stalls, but 
actually executed at 8 CPI. It then gives, for each instruction: 

■ the instruction address 

■ the source-code line number 

■ the opcode and operands 

■ the best-case number of cycles (which might be zero, for instruc¬ 
tions eligible for multiple-issuing) 

■ The estimated actual number of cycles (which also might be zero). 


** Best-case 4/6 = 0.67CPI, Actual 48/6 = 8.00CPI 


*** (8% execution without dynamic stalls) 


015798 1011: ldl 

t5,156(t2) 

1 

l.Ocy 

01579c 1013: and 

tO, Oxlf, a2 

0 

l.Ocy 

0157a0 1013: bis 

zero, 0x1, a3 

1 

l.Ocy 

0157a4 1013: sll 

a3, a2, a2 

1 

l.Ocy 

0157a8 1011: cmple 

t5, t6, t5 

1 43.5cy 

0157ac 1011: beq 

t5,0xl200157c0 

0 

0 


Figure 1. DCPI analysis of the worst basic block in comm_select (). 


Total CPU % 

Non-idle CPU % 

Procedure 

Mode 

65.80% 


all idle time 

kemd 

34.20% 

100.0% 

all non-idle time 


10.30% 

30.12% 

comm_select 

user (Squid) 

8.42% 

24.63% 

all select procedures 

kernel 

1.23% 

3.60% 

all TCP procedures 

kernel 

1.20% 

3.52% 

malloc-related #1 

user(libc) 

1.12% 

3.28% 

malloc-related #2 

user (libc) 

0.70% 

2.05% 

injpcblookup 

kemd 

0.60% 

1.76% 

malloc-related #3 

user (libc) 

0.45% 

1.31% 

malloc-related #5 

user (libc) 

0.32% 

0.95% 

bcopy 

kemd 

0.29% 

0.84% 

read_io_port 

kemd 

0.28% 

0.81% 

syscall 

kemd 

0.27% 

0.80% 

reg_save 

kemd 

0.27% 

0.80% 

memset 

user (libc) 

0.27% 

0.79% 

malloc-related #4 

user (libc) 

0.24% 

0.71% 

harddock 

kemd 

0.24% 

0.70% 

_dopmt 

user (libc) 

0.23% 

0.69% 

memCopy 

user (Squid) 

0.20% 

0.60% 

„XentInt 

kemd 

0.18% 

0.53% 

strcspn 

user (libc) 

Profile on 1998-07-16 from 11:00 to 12:00 PDT 

mean load = 17 requests/sec; peak load ca. 73 requests/sec 

Table 1. DCPI profile of unmodified Squid 
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1008 

1009 

>1010 

>1011 

1012 

1013 

1014 

1015 

>1016 

1017 

1018 

1019 

1020 


Note that the cmple instruction takes, on average, 43.5 cycles. Dcpicalc, in cryptic 
annotations not shown in Figure 1, also tells us that the most likely cause of this delay 
is a D-cache miss, that the instruction is probably waiting for data loaded from memo¬ 
ry by the ldl instruction, and that this instruction was generated from source line 
1011. 

Dcpicalc shows similarly lengthy delays at source 
lines 1010 and 1016. Now we know the location of 
the problem (or at least a problem): lots of D-cache 
misses at source lines 1010, 1011, and 1016 in 
comm_select (). Figure 2 shows that neighborhood 
of the source code. 

From this code excerpt, its pretty clear what is hap¬ 
pening. Comm_select () is building up two bitmaps 
for use as input to the select () system call, by iter¬ 
ating over an array of structures, one entry per file 
descriptor. The loop looks at three fields in each 
array entry, and is getting a D-cache miss on many 
(or perhaps all) of these references. fd_table is 
declared so that the three fields used in one iteration 
of this loop are not adjacent, and might even lie on 
separate cache lines. 

Because the number of active file descriptors is 
roughly proportional to event rate, and the cost of executing this loop is proportional 
to the number of file descriptors, the cost per event is proportional to the event rate. 
This causes 0(N ) behavior, which prevents efficient scaling. 

Statistics from our proxy server show that the maxf d variable, the upper limit of this 
loop, can reach values of 2380 or more, so each complete execution of this loop incurs 
as many as 2380 x 3 = 7140 D-cache misses. No wonder comm_select () takes so 
much time. 

Fixing the Problems 

Since I’m basically lazy, my first attempt to eliminate these D-cache misses was to sim¬ 
ply change the order of the fields in the declaration of fd_table, so that the three 
fields referenced on each loop iteration were adjacent, and then recompile Squid. This 
had the advantage that it shouldn’t require any debugging, and because the Alpha 
21164A has 64-byte cache lines, it should result in just one cache miss per iteration. 

That modification reduced the cost of comm_select () to 14% of the nonidle CPU 
cycles (from 30%). The average CPI for this procedure decreased from 2.69 to 2.43, 
mostly by reducing the number of cache misses on source line 1016. The discrepancy 
between the small change in CPI and the somewhat larger improvement in total time 
may reflect variation in Squid’s request rate, which was significantly higher during the 
second trial. 

Making the three interesting fields adjacent in each array element does improve the D- 
cache miss rate, but because the array elements contain many other fields that are not 
accessed during this loop, the D-cache is still filled mostly with useless information. I 
tried another small modification, creating a separate array to hold just these three 
fields. I reasoned that this would make full use of the D-cache during the loop. The 
result was a dramatic decrease in the CPI of comm_select (), to 0.75, but only a small 
decrease in the non-idle CPU cycle utilization of comm_select (), to 11%. 


for (i = 0; i < maxfd; i+ + ) { 

/* Check each open socket for a handler. */ 
if (fd_table[i].read_handler) { 

if (fd_table[i].stall_until <= squid_curtime) { 
nfds++; 

FD_SET(i, &readfds); 

} 

} 

if (fd_table[i].write_handler) { 
nfds++; 

FD_SET(i, &write fds); 

} 

} 

Figure 2: Source-code excerpt from comm_select () 
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This was frustrating. I had been able to eliminate most of the delays directly associated 
with D-cache misses, but the procedure was still taking more of the CPU than anything 
except the kernel's select () implementation. I finally admitted that I would have to 
try an algorithmic change, at the risk of more extensive debugging. 

The goal of the loop in question is to create two bitmaps for use by select {), based on 
the state of the entries in the fd_table. I changed the code to keep the bitmaps as 
long-lived data structures, updating them whenever the fields in fd_table are updated. 
Because select () destroys its input bitmaps, one actually has to make a copy of each 
bitmap before calling select (). Even accounting for the cost of this copy, the size of 
the data referenced before each call to select () decreases by a factor of about 40 
(because the Alpha uses 64-bit pointers; on a 32-bit CPU, the decrease is by a factor of 
24). 

One complication is that the loop compares the stall_until field against the current 
time, so the readf ds bitmap cannot simply be updated just when the readjiandler 
field changes. However, these timestamps have a granularity of one second, and the 
comm_select () function already iterates over the entire fd_table once per second (to 
check for other timeouts), so it adds almost no overhead to also do the stalljuntil 
comparisons once per second. 

With this change made (and debugged), the mean CPI for comnuselect () declined 
again, to 0.61, and its fraction of the nonidle CPU cycles declined to just 3.7%. 

Further work with dcpicalc showed that most 
of the remaining cycles in comm_select () were 
being spent on the second line of the code 
excerpted (in a heavily abstracted form) in 
Figure 3. This loop is going through the 
bitmaps returned by select (), looking for any 
“set” bits, and calling the corresponding han¬ 
dlers. The fd_isset () macro, which extracts a 
single bit from a large bitmap, executes numer¬ 
ous instructions, and so this loop executes many 
instructions per bit. 

The bitmaps typically have only a few bits set, 
and so it is much more efficient to treat each of 
them as an array of words, then iterate over this 
array looking for nonzero words (those with at 
least one bit set). This effectively parallelizes the 
search, and uses fewer instructions per step, so 
the revised loop executes (on average) a small 
fraction of an instruction per bit. Once it dis¬ 
covers a nonzero word, a simple loop then finds the nonzero bits in this word, calcu¬ 
lates the corresponding bitmap index, and invokes the corresponding handler. 

With this change, comm_select () now uses only 0.54% of the nonidle CPU cycles. Its 
CPI increases somewhat, to 1.18, because the last change eliminated lots of “useless” 
instructions but still examines the same bitmaps (and so still has about the same set of 
memory references). In spite of this dramatic improvement, comm_select () still has 
0(N ) behavior; the size of the bitmaps it manipulates remains proportional to the 
request rate. However, the changes reduce the cost by such a large constant factor as to 
make it unlikely that the comm_select () would ever dominate system costs at any 
request rate. 


for (fd = 0; fd < maxfd; fd++) { 
if (!FD_ISSET(fd, fcreadfds) && !FD_ISSET{fd, &writefds)) 
continue; 

if (FD_ISSET{fd, fcreadfds)) { 
if (fd_table[fd].read_handler) { 

fd_table[fd].read_handler(fd, fd_table[fd].read_data); 

} 

} 

if {FD_ISSET{fd, fcwritefds)) { 
if (fd_table[fd] .write_handler) { 

fd_table[fd] .write_handler (fd, fd_table[fd] ,write_data); 


} 


} 


Figure 3. Another source-code excerpt from comm.seiect () 
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Having succeeded in 
reducing the profiled cost 
of comm_select () from 
30% of the non idle cycles 
to just 0.54%, my last 
task was to verify that this 
actually improves overall 
performance. 


Measuring the Results 

Having succeeded in reducing the profiled cost of comm_select () from 30% of the 
nonidle cycles to just 0.54%, my last task was to verify that this actually improves over¬ 
all performance. 

Recall that I have samples, taken every 15 minutes, of both the request rate and the 
CPU utilization. On separate days, I ran trials of the original Squid proxy and the 
modified software; each 24-hour period yielded 96 samples. Figure 4 shows both sets 
of samples. Idle-time measurements are shown using circles, and user-mode CPU 
time measurements are shown using squares. Samples from the trial with the original 
software are distinguished by open marks; samples from the trial with the modified 
software are shown using filled marks. 



Mean requests/second 


Figure 4. CPU costs as a function of request rate 

Figure 4 includes lines showing linear regressions for each sample set. Table 2 summa¬ 
rizes these regressions. They show that the modifications make a 45% reduction in the 
slope of the regression for user-mode time; that is, the modified Squid expends much 
less user-mode CPU time per request than the original version. The slope of the 
regression for kernel-mode time has also improved slightly, apparently because the 
reduction in user-mode D-cache misses has improved the cache performance for other 
parts of the system. Overall, the slope of the idle-time regression has improved by 
30%. In other words, for a given request rate, the modified software leaves more idle 
time and hence should support a higher request rate. 

The X-intercepts of the idle-time regression lines imply that the modified version of 
Squid, on this hardware, can support a 45% higher request rate than the original. (The 
X-intercept is the projected point where the idle time drops to zero.) However, this 
projection is misleading; the system can actually sustain much higher request rates. 

The peak one-second rate actually logged during the trial with the original version was 
about 107 requests/sec, and was about 139 requests/sec with the modified version. 

There’s a simple reason for the discrepancy between the measured peak rates and the 
projected X-intercepts. When the system is lightly loaded, events arrive rarely enough 
that select () typically returns with just one ready descriptor. At increased loads, 
when the system has no idle time and the software falls behind, select () will return 
multiple events per call. In this region, each call to select () is amortized over multi¬ 
ple events, so its cost per event is reduced. Because select () and/or comm_select () 
account for the majority of the CPU-time costs, this “batching” effect allows the system 
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to handle increasing load even after running out of idle time. 

Ultimately, however, the more efficient software can handle a higher 
peak load, and provides less latency at lower loads, than the original 
version. 

Next Steps 

The results of this case study suggest further steps to take. First, my 
patches to comm_select () will be integrated into a future release of 
Squid. 

After application of these patches, and the kernel modifications to 
improve the performance of select 0 [3], the profiles still show that 
select () consumes a lot of CPU time, with user-level malloc () not far behind. A 
forthcoming release of Squid maintains its own pools of allocated memory for fre¬ 
quently-allocated data types, and so should put a lot less pressure on the malloc () 
functions, but it might still be worthwhile to study this cost in more detail. 

Reducing the cost of select () while preserving its programming interface, beyond the 
kernel modifications we have already tried, may prove difficult. In another paper [2], 
Gaurav Banga, Peter Druschel, and I propose a new system call interface to replace 
select () for this kind of application; preliminary results from simulated-WAN testing 
suggest that this interface eliminates virtually all of the select () -related costs, and 
scales to arbitrary numbers of file descriptors. 

Conclusions 

I set out to discover something interesting about how Squid s CPU utilization scales 
with request rate, and why. By using the appropriate performance-debugging tools, I 
quickly discovered the cause of a significant inefficiency. After a few false starts and 
some additional tool use, I devised a relatively simple change to the Squid code. This 
eliminated most of the user-mode CPU time spent in the Squid program. More 
important, it increased the efficiency of the entire system, as measured by tests on a live 
system. 

Of course, not every program has a single procedure whose performance can be 
improved by a factor of 55 through a simple programming change. But many pro¬ 
grams, especially Internet servers in complex environments, have performance charac¬ 
teristics that are not necessarily obvious from their source code. Further, the influence 
of memory-system stalls is both especially critical on modern CPUs and especially dif¬ 
ficult to discern from the source. One can gain tremendous leverage over these prob¬ 
lems by using advanced performance tools, such as DCPI, that provide whole-system 
performance views and expose memory-system issues. 


Squid Version 

CPU mode 

Slope 

Correlation 

coefficient 

X-intercept 

Original 

User 

0.99 

0.97 


Modified 

User 

0.54 

0.80 


Original 

Kernel 

0.82 

0.98 


Modified 

Kernel 

0.73 

0.95 


Original 

Idle 

-1.81 

-0.98 

57.5 

Modified 

Idle 

-1.26 

-0.92 

83.4 

Table 2. Linear regressions of CPU costs as a function of request rate 
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using java 

The Java Native Interface 


In a previous ;logiri: article (June 1998), I discussed the embedding of a Java 
Bean into an ActiveX environment. I showed that components written in Java 
can interoperate with components written in other languages at the “compo¬ 
nent” level, namely ActiveX components. This article continues the theme of 
interoperability, but with a different perspective. We will look at the Java Native 
Interface (JNI), which is also a very important part of the Java development 
infrastructure. The JNI capability permits programs written in Java to call pro¬ 
grams written in C or C++, and also the reverse: a C or C++ program can invoke 
the Java Virtual Machine (JVM) and invoke methods on a Java object. 
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This capability is mandated in several situations. For instance, in order to run Abstract 
Windowing ToolKit (AWT) applications, the graphics context for the application has to 
eventually make “native” calls to the libraries that perform the graphics operations. It is 
likely that these libraries are written in C (more likely than C++). Also, Java communi¬ 
cation packages eventually make “native” calls; we can gather from this that “native” 
refers to either C or C++. Finally, it is likely that most organizations have significant 
amounts of C and C++ code that has been debugged and tested. JNI plays a vital role 
in supporting the use of this code. 

Lets start with a summary of the main considerations of JNI, then present examples of 
code and a more detailed description of this capability. 

Java Native Interface (JNI) 

The Java Native Interface is a standard programming interface between Java and native 
programs. Typically, native programs are used when they cannot be written in Java 
(legacy code, performance considerations). 

The JNI makes it possible to create, inspect, and update Java objects including arrays 
and strings. The JNI also permits the catching and throwing of exceptions, loads class¬ 
es, obtains class information, and performs runtime type checking. It is possible using 
JNI to link a Java Virtual Machine (JVM) into non-Java applications. 

As with any important technology, interested parties will jostle for acceptance of their 
ideas. Sun, with its JNI, is one of three currently trying to influence the Native Interface 
standard. The other two main parties who can influence the standard are Netscape with 
the Java Runtime Interface (JRI) and Microsoft with the Raw Native Interface (RNI). 
The standard is still evolving, and this article will discuss only JNI from Sun. 

The advantages of a standard are that each vendor can support a large body of native 
code. Also, tool builders need not maintain different kinds of native method interfaces. 
This means that application programmers write one version of their native code. Some 
requirements of the evolving standard are: 

■ Binary compatibility so that one version of native code exists. 

■ Efficiency so that the overhead of the interface should be small. 

■ Functionality so that native methods must be able to do useful things. 

It appears that JNI is becoming the standard, so it is probably a good idea to program 
to it. 
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Let’s take a look at an 
example of how JNI can be 
used to call C programs 
from Java. 


Working with Native Methods 

The steps involved in using JNI are: 

1) Write the Java code and define relevant methods “native.” 

2) Compile the Java code. 

3) Create a header file that acts as a bridge between Java and native code. 

4) Implement the native code. 

5) Create a shared library of the native code. 

JNI Example 

Lets take a look at an example of how JNI can be used to call C programs from Java. 
(In a future article I will discuss how to do the reverse.) We will look at a package to 
manipulate rational numbers. The package supplies the following routines: 

■ radd - add two rational numbers 

■ rsub - subtract two rational numbers 

■ rmul - multiply two rational numbers 

■ rdiv - divide two rational numbers 

■ euclid_gcd - the greatest common divisor of two integers 

We will create a Java class called Rational: 

public class Rational 
{ 

public int[] r; 

public Rational(int n, int d) 

{ 

if (d == 0) 

throw new ArithmeticException("Rational: requires " + 

"non-zero denominator") ; 

r = new int[2]; 
r[0] = n; 
r [1] = d; 

} 

public static Rational RationalAdd(Rational rl, Rational r2) 

{ 

return new Rational(add(rl.r, r2.r)); 

} 

public static Rational RationalSub(Rational rl, Rational r2) 

{ 

return new Rational(sub(rl.r, r2.r)); 

} 

public static Rational RationalMul(Rational rl, Rational r2) 

{ 

return new Rational(mul(rl.r, r2.r)); 

> 

public static Rational RationalDiv(Rational rl, Rational r2) 

{ 

return new Rational(div(rl.r, r2.r)); 

} 
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Next we define the native methods: 

public static native int[] add(int[] rl, int[] r2); 

This says that there is a function called “add” which is not written in Java, 
public static native int[] sub(int[] rl, int[] r2); 

public static native int[] mul(int[] rl, int[] r2); 

public static native int[] div(int[] rl, int[] r2); 

public static native int[] gcd(int x, int y); 

Next we create a toString method. This is always a good idea: 

public String toString() 

{ 

StringBuffer sb = new StringBuffer{); 

sb.append(r[0]); 
sb.append(’/'); 
sb.append(r [1]) ; 

return new String(sb); 

} 

Now we need to add code to load the shared library: 

static 

{ 

System.loadLibrary("rat"); 

} 

This call loads the librat. so (Solaris) shared library so that Java programs can invoke 
the native calls. 

Now that we have the Rational class defined we can compile the code: 
javac Rational.java 

This produces a .class file from which we create a header file which acts as a “bridge” 
between the Java and C code, 
javah -jni Rational 

The output of this is a file “Rational.h”. The entries in this file must not be edited, 
because the JVM uses it to interpret the parameters being passed between Java and the 
native code. 

For our native function “add” the entry in this file looks like this: 

/* 

* Class: Rational 

* Method: add 

* Signature: ([I [I)[I 
*/ 

JNIEXPORT jintArray JNICALL Java_Rational_add 
(JNIEnv *, jclass, jintArray, jintArray); 

Every native function will have a similar entry based on its signature. We will discuss 
the details in a subsequent article. 

Next we write the C code that implements the native functions. We include only the 
“add” example here. Readers can send mail to <prithvi+@kiwilabs.com> for a full code 
example. 
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It is entirely up to the 
programmer to guarantee 
that the native call is 
thread-safe. 


/* 

The interface between java and C for the rational number package 
*/ 

#include <jni.h> 

#include "Rational.h" 

#include <stdio.h> 

int *radd(int *, int *); 

JNIEXPORT jintArray JNICALL 

Java_Rational_add(JNIEnv *jenv, jclass jc, jintArray jrl, 
jintArray jr2) 

{ 

jint *r; 
jintArray jr; 

jsize rlen = (*jenv)->GetArrayLength(jenv, jrl); 

jint *rlarr = (*jenv)->GetIntArrayElements(jenv, jrl, 0) ; 

jint *r2arr = (*jenv)->GetintArrayElements(jenv, jr2, 0); 

(*jenv)->ReleaseIntArrayElements(jenv, jrl, rlarr, 0); 

(*jenv)->ReleaseIntArrayElements(jenv, jr2, r2arr, 0); 

r = (jint *) radd((int *) rlarr, (int *) r2arr); 
j r = (* j env)->NewIntArray(j env, rlen); 

(*jenv)->SetIntArrayRegion(jenv, jr, 0, rlen, r) ; 

return jr; 

} 

The above code example is the native code implementation of “add”. In subsequent 
articles I will cover in more detail the writing of native functions. For now it is impor¬ 
tant to note that native code includes the “jni.h” and “Rational.h” include files. These 
two files are crucial because they form the “bridge” which permits the Java native pro¬ 
grams to pass parameters between them. 

JNI and Threading 

Since Java is a multithreaded language, several threads can call a native method concur¬ 
rently. However, it is possible that a native method might be suspended in the middle 
of its operation when a second thread calls it. It is entirely up to the programmer to 
guarantee that the native call is thread-safe. One way of doing this is to declare the 
native thread as “synchronized” or implement some other strategy within the native 
method to ensure correct concurrent data manipulation. 

Another consideration of multithreading is that the JNIEnv pointer must not be passed 
across threads. This is because the internal structure to which it points is allocated on a 
per-thread basis and so contains information that makes sense only in that particular 
thread. 

JNI and Java Exceptions 

With JNI, Java exceptions can be thrown, caught, printed, and rethrown just as they are 
inside a Java program. But its up to the programmer to call dedicated JNI functions to 
deal with exceptions. Some JNI functions for exception handling are: 

■ Throw (): Throws an existing exception object. Used in native methods to rethrow an 
exception. 

■ ThrowNew(): Generates a new exception object and throws it. 

■ ExceptionOccurred (): Determines if an exception was thrown and not yet cleared. 

■ ExceptionDescribe (): Prints an exception and the stack trace. 
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■ ExceptionClear (): Clears a pending exception. 

■ FatalError (): Raises a fatal exception. 

Of these exceptions, it is not possible to ignore ExceptionOccurred () and 
ExceptionClear ( ). After exceptions are handled by Except ionOccur red ( ) it is nec¬ 
essary to clear them using ExceptionClear () to avoid unpredictable results (especially 
if a call is made to a JNI function while an exception is pending). 

Conclusion 

We have looked at the use of JNI for calling native methods written in C from a Java 
program. Clearly, this is a very important consideration in Java being a credible lan¬ 
guage; Java could not be taken seriously were this not possible. It is equally important 
to be able to invoke Java methods from native code, and this is also possible. 

A strength of Java is that not only is native code integration possible, but it is simple 
and straightforward with all the tools available as part of the JDK distribution. 

However, it is still necessary to understand the use of JNI within the larger context of 
multithreading and exception handling, which have important implications when you 
are working with native interfaces. 
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This interview with Barb Dijker was con¬ 
ducted by Rob Kolstad just prior to LISA 
and its Great Certification Debate. Barb 
has been active in getting the certification 
effort off the ground. 


The Great Certification 
Debate 

Interview with Barb Dijker 

Rob: How did you become involved in the certification effort? 

Barb: The same way I became a system administrator. Its something that appears to 
need doing, someone needs to do it, and it wasn’t getting done. I didn’t start this, nor 
did I push it through the SAGE Executive. This is a significant undertaking with broad 
implications. I’m one of four on the SAGE certification subcommittee. Another 48 
SAGE members are on an advisory council - which will hopefully keep us from screw¬ 
ing up. 

Rob: Are you personally for or against certification? 

Barb: Personally I don’t have strong inclinations one way or the other. The reason is 
that I don’t think we really have enough data yet to substantiate arguments on either 
side of the issue. That’s why this effort is important. The initial goal is to get the data so 
we can make a decision. 

Rob: And what is the final goal of SAGE certification? 

Barb: Once a certification program is developed, its purpose would be to provide an 
objective means of skill assessment. It would be a guide for new system administrators 
to develop their skills and a means for hiring managers to wade through resumes. 

Rob: Does anyone really want that? 

Barb: Employers eat it up. Those looking to get into the high-salary world of system 
administration would have a road map to do it. I’m serious. To a PC repair person, sys¬ 
tem administration is “high-salary” even if certification does water down salaries as 
doomsayers claim it will. Anyone with good problem-solving skills can learn to be a 
system administrator. We need more system administrators. How else are we going to 
grow them? 

Rob: Shouldn’t we be helping universities build curricula instead? 

Barb: In addition to, but not instead. Getting a degree in computer science, even if 
there were a minor in system administration, isn’t going to necessarily teach you every¬ 
thing you need to know about doing it on the job. Nor does it tell a potential employer 
how much of that curriculum soaked into your brain and contributed to your actual 
skill level. If higher education were the only answer, there would be no certification 
programs anywhere. 

Rob: How do you implement it? 

Barb: SAGE has never done anything this big. So implementation is as much of the 
problem as determining the required skill sets and developing methods to assess them. 
However, there are many professional certification-program development and execu¬ 
tion companies that can help us. SAGE isn’t the first professional or trade association to 
want to provide a certification program for its members. Think of teachers, nurses, 
cops, etc. All of them get “professional” outside help to develop their programs. As sys¬ 
tem administrators, we want our users to recognize when they should call in a “profes¬ 
sional” rather than taking the do-it-yourself approach. By the same token, we need to 
recognize that developing a certification program is completely outside of our skill set. 
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Rob: This sounds really expensive. How can it be worked out so it’s affordable to those 
being certified? 

Barb: It may turn out that it is cost-prohibitive. We don’t have all the data. At this 
point, we aren’t developing a certification program yet. We are only doing a feasibility 
study to justify a recommendation of whether to develop and implement a certification 
program or not. Part of the criteria for the recommendation is that the program must 
be affordable. If it isn’t affordable to the participants, then it doesn’t have enough value. 

On the other hand, SAGE isn’t doing this to make a profit like a company. So the pro¬ 
gram only needs to cover the cost of implementation and maintenance. 

Rob: Is everyone in favor of this? 

Barb: That’s impossible to say. Everyone on the SAGE Executive was in favor of this 
effort - or at least not opposed to it. When SAGE conducted a survey last year, there 
was 2:1 support. However, we know that the survey results do not statistically represent 
the SAGE membership or the greater system-administrator community. It’s just an 
interesting datapoint - a conversation starter. And boy did it stir up conversation. 

When the SAGE Executive first announced the plan to start down this road back in 
February, the <sage-members> mailing list was flooded with a heated debate. A certifica¬ 
tion program isn’t any good unless it is accepted by a reasonable majority of the mem¬ 
bership. That’s why a major aspect of this effort is to open constructive dialogue about 
certification. A Web site, <http://www.usenix.org/sage/cert/>, is online for information and 
feedback and debates, and BOFs are being conducted. 

Initially the feedback was negative. This was mostly based on fear that SAGE would re¬ 
create MSCE - which is generally viewed by SAGE members as an expensive waste of 
time - or that SAGE would mandate certification for membership or try to certify 
senior sysadmins. Since those fears have been dispelled, most of the feedback has been 
a very positive and “it’s about time” sort of thing. 

Rob: Do I need to keep getting recertified every year or two or keep getting add-on 
credits or something? 


A certification 
program isn’t 
any good unless it is 
accepted by a reasonable 
majority of the member¬ 
ship. That’s why a major 
aspect of this effort is to 
open constructive 
dialogue about 
certification. 




Barb: That has not yet been determined. The initial focus is on “core competency.” 

That basically means certification for a solid entry-level system administrator. However, 
another goal is to design the program so that it is extensible. So in the future there 
might be a “postmaster” certification, for example. A moving target like that would 
need recertification, or at the very least a date stamp, to be meaningful. 

Rob: Thanks! 
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I just got back from the LISA conference in Boston. Boston was okay: not as 
cold as I had feared, plenty of good restaurants, and lots of people did show 
up. I think it was one of the largest LISA conferences ever, actually. While 
standing facing a wall, as men often do, an exhilarated attendee told me that 
he got more from standing in the halls talking with people than at many other 
conferences he had attended. I can’t say that I was surprised. 

Of course, there were also the traditional activities, such as listening to papers or invit¬ 
ed talks, the LISA quiz show, and drinking beer (thats why we were standing at the 
wall). 

Eric Allman, author of sendmail and a genuine USENIX hero, gave the keynote. Eric 
has decided that it was high time he made some money from sendmail. He once told 
me that he wished that he had a dime for every copy of sendmail that was in use. He 
may do better than that for some of the copies out there, once the commercial version 
of sendmail appears. But sendmail will remain Open Source (a trademark of the Open 
Source Initiative; see <www.opensource.org>). 

There was a time when I would have defined the USENIX Association as the largest 
organization founded on the idea of free software. The heroes of USENIX, like Eric, 
produced useful tools or programs and made not only the compiled programs but the 
source to the programs available as well. Revealing the source has several side effects, 
not the least of which being what happens when many other programmers start look¬ 
ing over your code. This is not for the faint of heart, or the truly arrogant either. 
Instead, it is both a humbling and a useful experience. People will find fault with either 
your coding or your designs (and often both), but in the end the code gets better. 

Opening Remarks 

Eric began the keynote by describing the history of software as a three-act play. The 
first act included mainframes and minicomputers. During the question-and-answer 
session that followed, Peter Salus pointed out that open source had its roots in SHARE, 
when '50s (and later) mainframe programmers shared source code on punched cards. 
But that era is best described as closed and proprietary, with large expensive computers 
in glass rooms (think of glass display cases) and a priesthood to minister to their needs. 
Digital Equipment s PDP series did make computing more accessible (computers in the 
tens of thousands of dollars instead of millions), but the operating systems and soft¬ 
ware were still closed. 

Act 2 in this chronology was the advent of first the homebrew, then the personal com¬ 
puter. Both the homebrew community and the early Apple computers (I and II) spon¬ 
sored an “extend and return” mentality. Programmers would generally share source and 
communicate any extensions or changes made back to the original author. Note that 
Bill Gates began Microsoft by not doing this. He and a few programmers created a pro¬ 
prietary version of what had been a free version of BASIC and began selling it. Their 
extensions were never returned to the community, which was the original reason for 
the animosity toward Microsoft. 

Act 3 opens with the beginning of virtual open source - UNIX and the Berkeley 
Software Distribution, or BSD. The UNIX system was actually proprietary, but because 
of the actions of AT&T, many schools and universities had access to the source code. 
BSD included many extensions to the standard UNIX distribution and was controlled 
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by the Computer Science Research Group (CSRG) at the University of California, 
Berkeley campus. 

Bill Joy, better known as one of the four founders of Sun Microsystems, crafted the 
early releases of BSD. Not only did he contribute code (vi and csh, for example), he 
also decided what software would be included in each release. 

Eric pointed out that Samba, FreeBSD, NetBSD, OpenBSD, Linux, Apache, BIND, and 
sendmail all share a similar feature - that of a central, controlling person or group. 
Without some form of release management - deciding what features and patches will 
make up a release - software tends to fragment. Core teams watch over the BSDs, 

Linux, Apache, Samba, and BIND, while Eric watches over sendmail. In this, Eric dis¬ 
agrees with Eric Raymonds “The Bazaar and the Cathedral” paper (<http://www.tuxedo.org/ 
~esr/writings/cathedral-bazaar/>), which describes a somewhat different model of open- 
source creation. 

Money 

Open Source also means the birth of hybrid organizations, such as Red Hat or Erics 
Sendmail, Inc. These organizations still give away the source but make money while 
doing so. Red Hat makes money by selling CDs and documentation and by providing 
handholding. Sendmail, Inc. will make money by selling extensions, such as an easy-to- 
use configuration GUI, as well as commercial support. 

If you visit the Open Source Web page, you will quickly discover that the difference 
between “free software” and open source is largely one of marketing - but not totally. 
Marketing, because you cannot sell “free software.” And not totally different, because 
open-source software will be freely available, although covered by license agreements. A 
lot of the Open Source Web site is devoted to explaining how open-source software can 
be a good thing for programmers. 

And big business as well. IBM Research released Jikes, a Java compiler, as open source 
the first week of December. Earlier in this year, IBM also announced that it would offer 
support for the Apache Web server. If IBM can make money supporting software that 
they did not create and do not own, why shouldn’t it? Service and support have always 
been a large part of computer companies’ income. While IBM’s support helps to legit¬ 
imize open-source software, it also places IBM in competition with non-IBM program¬ 
mers and consultants. At this point, I think IBM’s recognition is more important than 
possible competition. 

Speaking of competition, one of the LISA invited talks set the authors or representa¬ 
tives of four MTA (Mail Transport Agents) head-to-head in a panel. Besides Eric 
Allman, there was Wietse Venema, who had just completed Postfix (previously Vmailer; 
see <www.ibm.com/alphaworks>), Dan Bernstein, author of qmail (<www.qmail.org>), and a 
representative of the University of Cambridge, author of exim. Interestingly to me, all 
three replacements for sendmail claim better security as one of their main goals. But 
sendmail has had a remarkable security track record recently ; leaving security as per¬ 
haps a poor method for choosing an MTA. Visit the Web sites (including 
<www.sendmail.org>) and choose your own favorite - if you haven’t already done so. 

Dan Klein repeated his talk at LISA on the “Succumbing to the Dark Side of the Force: 
The Internet as Seen from an Adult Website.” Although you might have expected Dan’s 
talk - which does include lots of useful Web-server lore and little about the other side 
of the business - to have drawn the entire conference attendance, the papers track, 


Eric pointed out that 
Samba, FreeBSD, NetBSD, 
OpenBSD, Linux, Apache, 
BIND, and sendmail all 
share a similar feature - 
that of a central, 
controlling person or 
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February 1999 ;login: 


67 


FEATURES 







I quickly learned how 
wrong I was to assume 
that a head-mounted 
display should be much 
less power hungry and 
easier to support than a 
large notebook display. 


including a talk by Chris Page about configuring database servers, drew a large crowd 
as well. 

The Borg 

I had hoped that while I was in Boston I could slip away from the conference and meet 
with some of the wearable-computing folk, occasionally known as the borg, over at the 
MIT Media Lab. I had not counted upon the end of the semester, as well as a thesis 
defense, occurring at the same time. I did get to meet with Dave Kaplowitz (Dkap), 
now at Tufts University, who showed me around the Media Lab Friday afternoon. You 
may have seen Dkap at the last LISA conference, in San Diego, and he was also in atten¬ 
dance in Boston. His rig includes a Private Eye display, a Twiddler for input, a box 
containing an Intel 486 processor and hard drive, and a fairly heavy set of lead-acid 
batteries. 

When we met at MIT, Dkap had left his wearable at Tufts. There had been a security 
incident, and he had left his wearable behind working on a Tripwire check of the invad¬ 
ed system. The lead-acid batteries, although bulky, do provide all-day power and make 
a useful system for Dkap in his system-administration duties. It did seem strange to see 
him without his wearable. 

I also had a chance to talk with Steve Schwartz, a research scientist hired specifically to 
work on wearables at the Media Lab. Steve has worked with head-mounted displays 
(HMDs) for many years, starting with work with Lucas Films in the ’80s. Steve provid¬ 
ed me with a rapid-fire history of HMDs, and I quickly learned how wrong I was to 
assume that an HMD should be much less power hungry and easier to support than a 
large notebook display. HMDs can be less power hungry, but getting them to work 
requires special hardware - they do not work directly from VGA output, any more than 
the new flatscreen display monitors do. What’s worse, companies that have focused on 
producing HMDs have had a bad habit of either going out of business (as Private Eye 
has) or focusing on other products. 

As I wandered around the Media Lab, I also learned that the wearables group 
(<www.mit.edu/wearables>) is spread throughout the building and organization. For most 
people in the Media Lab, a wearable computer is not a goal, but rather a tool for 
exploring new applications made feasible by the wearable. Software agents that adapt to 
what you are doing as well as your past behavior. Effective computing, getting comput¬ 
ers to recognize and respond to human moods. Not being strange, borglike humans. 

I plan on revisiting the topic of wearables in the not-too-distant future. I am astonished 
by the number of people in the USENIX community who are already using wearables - 
mostly PalmPilots, with a scattering of other types of what are today called handhelds. 
These computers are worn until they are needed, then they are opened, turned on, then 
put back into a pocket until needed again. Not quite true wearables, but getting close. 
And the age of true wearable computing is not quite here yet, either. 
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The Single European Currency 
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Abstract 

By now we are aware of the Year 2000 
problem. However, enterprises doing 
business in or with European counties 
have to contend with another major soft¬ 
ware issue, the introduction in January 
1999 of a new single European currency 
called the euro. This article is a brief 
introduction to the euro and the impact 
that its introduction will have on operat¬ 
ing systems and applications. 

Introduction 

In 1998, the third stage of the Economic 
and Monetary Union (EMU) was 
launched with the establishment of a 
European Central Bank (ECB) in 
Frankfurt, Germany, to manage the intro¬ 
duction of a new single pan-European 
currency. The name chosen for the basic 
denomination of this currency was the 
euro. 

The introduction of the euro will be one 
of the major economic events in the next 
decade. A common currency will con¬ 
tribute significantly to the further eco¬ 
nomic integration of the participating 
countries. It will increase market trans¬ 
parency by making prices more easily 
comparable. Pan-European trade will 
become more attractive as trade and 
investment will no longer be exposed to 
exchange-rate risks, and the costs associ¬ 
ated with currency conversion will be 
eliminated. Eleven countries qualified for 


initial membership in the EMU: Austria, 
Belgium, Finland, France, Germany, 
Ireland, Italy, Luxembourg, Portugal, 
Spain, and the Netherlands. 

On January 1, 1999, the rate of conver¬ 
sion between the euro and the currencies 
of the eleven participating countries will 
be irrevocably fixed, and the ECB will 
begin the administration of a single 
monetary policy. On this date, the euro, 
in non-cash form, will become legal ten¬ 
der in these countries. This means that 
the euro will be usable for non-cash 
transactions such as checks and credit 
transfers. 

On January 1, 2002, new euro banknotes 
and coins will be put into circulation. 
Over a period of time, determined by 
each participating country, notes and 
coinage in the national currency will be 
withdrawn from circulation. By July 1, 
2002, euro notes and coins will be the 
only legal tender in the participating 
countries. 

Currency Symbol 

The European Commission has intro¬ 
duced a new currency symbol to repre¬ 
sent the new currency and has registered 
the currency code EUR with the ISO 4217 
(Currency Codes) Maintenance Agency 
as representing this currency. 

The new currency symbol is essentially a 
three-quarters circle with two horizontal 
crossbars, as shown below: 



Support for the euro currency symbol in 
computer systems and applications 
means the ability to input, display, print, 
store, and retrieve the symbol. This 
requires a small number of changes to 
operating systems and various interna¬ 
tional standards. 

The Unicode Standard and ISO/IEC 
10646-1:1993 (UCS) included a symbol 
called the Euro-currency sign in the 
block of currency symbols at code posi- 
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tion 0x20a0, and a corresponding glyph “CE” consisting of an 
interleaved “C” and “E”, with the “E” being lower and more to 
the right than the “C”. The meaning of this symbol was recently 
clarified and declared to represent the ECU (European Currency 
Unit).The ECU is an existing, mainly paperless, currency used 
within the European Union. Its value is based on a basket of 
national currencies. One ECU will be equivalent to one euro. 
Upon the introduction of the euro, use of the ECU will be 
phased out. 

In order to represent the euro currency, a new symbol called 
“EURO SIGN” was added to ISO/IEC 10646-1 in the block of 
currency symbols at code position 0x20ac, and a new glyph rep¬ 
resenting this symbol was registered in the ISO/IEC 10036 Glyph 
Registry. 

Images of the EUROCURRENCY SIGN and EURO SIGN 
glyphs, and the new notes and coinage may be viewed at 
<http://www.euro.eu.int/euro/html/entry.html>. 

The Final Committee Draft ISO 8859-15 (Latin9) standard 
(nicknamed LatinO because it updates Latin 1 with some forgot¬ 
ten French and Finnish characters) replaces the code-point for 
the generic international currency symbol (0xa4) with the euro 
currency symbol. 

The HTML 4.0 specification defines the entity “&euro;” 
(“8c#8364;”) as the euro currency symbol. 

Support for the euro is planned in the next major release (VI.2) 
of the Java Development Kit (JDK). Before then, support will be 
provided in a maintenance release (VI. 1.7) of the JDK because 
of the immediate need for such support. The specification of the 
java. lang.Character class was updated to include the euro 
currency symbol. 

The XI 1R6.4 Sample Implementation included support for the 
euro currency symbol in Public Patch Number 2. The € symbol 
is defined in <Xll/keysymdef .h>. However, many of the fonts 
in the Sample Implementation use IS08859-1 encoding, which 
does not include support for the euro currency symbol. 

A large number of font vendors have produced new fonts which 
include the € glyph. For example, Adobe Systems has developed 
a number of Type 1 Postscript fonts. These are available for free 
download at <http://www.adobe.com/type/eurofont.htmI>. 

Most operating-system vendors are planning to include at least 
basic support for the new currency in their next major release. 
Typically this support will include the following: 

■ A set of UTF-8 locales which will include support for the € 
symbol 

■ Extensions to the X Window System font library to enable dis¬ 
play of these locales 


■ Modification of locale database and resource files to support 
the new UTF-8 locales 

■ Input method and printing support for the € symbol as 
required 

The complete set of UTF-8 locales will probably not be supplied 
in the first release containing support for the euro currency sym¬ 
bol. Expect to see UTF-8 locales for those countries that are part 
of the European Monetary Union first. 

Impact on Applications 

The impact of the introduction of the euro currency is essential¬ 
ly a software issue - only systems that process financial informa¬ 
tion in one of the participating national currencies are affected 
by the euro changeover. 

For organizations doing business in any of the participating 
countries, modifying applications to accommodate the euro is of 
the highest priority - probably even more than the Year 2000 
problem because of the January 1,1999, introduction date. The 
Year 2000 problem is basically a technical problem, whereas the 
euro currency changeover requires additional functionality in 
applications. 

Where the impact of the euro currency introduction will be felt 
most is in large multinational companies, stock and bond mar¬ 
kets, and all levels of the financial markets. This will affect non- 
European banks, financial institutions, and businesses also. 

Examples of applications that are affected include: 

■ Accounting systems 

■ Invoicing and billing systems 

■ Payroll systems 

■ Fixed asset systems 

■ Financial planning systems 

■ Enterprise resource planning systems 

■ Treasury management systems. 

Generic software such as Microsoft Office and Lotus SmartSuite 
will also have to be upgraded. Many spreadsheets dealing with 
financial information will have to be redesigned. 

Conversion between the euro and a participating national cur¬ 
rency will cause a rounding error and a loss of precision. Specific 
legal requirements for expressing amounts in euro and for con¬ 
verting between national currencies have been established, and 
applications will have to be modified to ensure that these 
requirements are fully implemented. Of particular significance 
are rounding errors caused by cumulative data operations. 
Conversion between two national currencies must be via the 
euro. The use of inverse exchange rates to convert amounts 
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between a national currency and the euro is not permitted. 

Many existing applications convert currencies using cross 
rates and inverse rates. Such applications will require extensive 
modification. 

Applications that were designed to work with a national curren¬ 
cy with no decimals will need to expand the currency field to 
work with decimals, since the euro currency includes cents (100 
cents to the euro). Such countries include Spain, Belgium, and 
Italy. 

It may be difficult to display the euro and a national currency 
simultaneously, as required during the transition period 
(January 1999 to mid-2002), when both the euro and national 
currencies are valid legal tender (dual currency), because of a 
shortage of screen and printed report space. Some applications 
may require a major redesign to comply with this requirement. 

Many applications use some sort of threshold value to define 
various actions. For example, a business may have one shipping 
charge for orders under $500.00 and a different charge for orders 
over $500.00, with the application automatically calculating the 
shipping charge. Such thresholds will have to be identified and 
changed. Other common thresholds used in applications include 
authorization levels, data validity checking, and exception report 
generation. 

It may not be possible to change all applications over to the euro 
at the same time. This means that applications will have to com¬ 
municate with applications that are not euro-compliant. Many 
enterprises have links to systems belonging to other enterprises. 
In this case, the changeover to the euro requires careful coordi¬ 
nation between the various enterprises. 

At some time, enterprises will have to convert over to the euro 
currency completely. When this occurs, a considerable amount 
of historical data will probably also have to be converted from 
the national currency to the euro. 

Keyboards 

Keyboard standards distinguish three main levels of 
functionality: 

■ Level 1 - Press the “m” key to produce the character “m”. 

■ Level 2 - Press the Shift key and the “m” key simultaneously to 
produce the uppercase character “M”. 

■ Level 3 - Press the AltGr key and the “e” key simultaneously to 
produce the € symbol. 

The European Commission is determined that the euro currency 
symbol € will have a prominent, i.e. visible and easy to use, 
placement on user's keyboards, and has issued two proposals. 

The short-term proposal is for all Latin and Greek alphabet- 
based keyboards to place the euro currency symbol on the “E” 


keycap (key D 03 on Level 3). Simultaneously pressing the 
“AltGr” (The Right Alt key) and “E” keys generates the euro cur¬ 
rency symbol. Many existing European keyboards use this type 
of engraving. This solution is common to all major types of key¬ 
boards, is simple to implement, and is easy to remember because 
of the association of E with the euro. However, this proposal 
causes problems for written Irish (Gaeilge) and some other lan¬ 
guages. A number of vendors are using AltGr+4 for Irish and 
United Kingdom keyboards, AltGr+5 for Greek Latin and 
Hebrew keyboards, and AltGr+u for Hungarian and Polish key¬ 
boards. On keyboards with no AltGr key, the euro currency sym¬ 
bol may be produced by the sequence Ctrl+Alt+E. 

A new draft of the ISO/IEC 9995-3 keyboard standard is being 
balloted. This includes the € in the common secondary layout 
and thus provides for the placement of the € in those keyboards 
where the primary layout does not include the €. 

The long-term proposal is to introduce a new keycap which will 
be placed in the same location on all major keyboard layouts. 
This would be a level 1 key. The proposed location is under the 
Carriage Return key, to the left or right of the Shift key located 
right under the Carriage Return key. A number of vendors have 
already introduced keyboards conforming to this proposal. 

Printers 

The way the euro currency symbol is handled by a printer will 
depend upon the operating system and the particular printer 
capabilities. Since the majority of existing printers do not have 
built-in support for the euro currency symbol (i.e., in their resi¬ 
dent fonts) most operating system vendors will deploy a soft¬ 
ware approach using a downloadable font. In essence, the euro 
currency symbol will be treated as a small graphic in a font 
which can be downloaded to the printer whenever the euro cur¬ 
rency symbol is to be printed. Printing fonts as graphics instead 
of using a resident font may result in slower printing. 

Many font vendors have made the new euro currency symbol 
available either as a standalone glyph which precisely matches 
the definition of the symbol as published by the European 
Commission, or as a glyph that is built into a typeface library 
and which matches the characteristics of a specific font. This 
arrangement gives users the choice of using either the precise 
symbol or a modified symbol which matches their favorite font. 

By mid-1999, expect most new printers to have support for the 
euro currency symbol built into their resident fonts. 

Further Information 

The following documents are available for download at 
<http://www.ipso.cec.be/y2keuro/euroit.htm>: 

■ Preparing Financial Information Systems for the Euro, 
12/15/97, European Commission, Brussels 
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■ Recommendation for the Placement of the Euro Sign on 
Computer Keyboards and Similar Information Processing 
Equipment, VI.5 (6/19/1998), European Commission, 
Brussels. 

■ The Introduction of the Euro and the Rounding of Currency 
Amounts, Europaper N.22. 

■ Preparing Information Systems for the Euro, 9/25/97, 
European Commission, Brussels 

POSlX.lh SRASS and POSlX.lm Checkpoint/Restart 

Helmut Roth <hroth@nswc.navy.mil> reports on the October 1998 
meeting in San Diego , CA. 

The POSlX.lh Services for Reliable, Available and Serviceable 
Systems (SRASS) and the POSlX.lm Checkpoint/Restart work¬ 
ing group met in San Diego, California, in late October 1998. 
The SRASS working group is in the process of developing a set 
of APIs for fault management and serviceability applications. 
The goal of the SRASS Working Group is to support fault-toler¬ 
ant systems, serviceable systems, reliable systems, and highly 
available systems in a portable way. Wherever feasible, POSlX.lh 
needs to be useful for general applications too, such as distrib¬ 
uted parallel database transaction systems and safety-related 
systems. 

Checkpoint/Restart allows an application to save the entire state 
of the machine along with the operating system and the process 
activities so that in the event that something goes wrong, a saved 
backup state can be brought on line quickly. A ballot group has 
been formed, and the Checkpoint/Restart API should go out for 
ballot in November 1998. Some minor corrections to the draft to 
support a more rigid set of rules for use of the namespace and 
other backward-compatibility issues are being added. It is 
intended to be out to the ballot group in time to ballot before 
the next meeting in January 1999. 

One part of the SRASS draft deals with logging APIs. These are 
aimed at allowing an application to log application-specific 
events and system events to a system log and allow for the subse¬ 
quent processing of those events. Fault-management applica¬ 
tions can use this API to register for the notification of events as 
they enter the system log. An example of an event could be 


where some limit has been exceeded. Events can have a severity 
associated with them. Event notification can provide a way to 
react proactively and initiate steps to prevent a subsequent sys¬ 
tem failure. In addition to these logging APIs, the de facto stan¬ 
dard syslog interfaces have been added to support backward 
compatibility. 

Another feature of the SRASS proposed interfaces is a single 
core-dump-control API. This is intended to enable an applica¬ 
tion to specify the location of a file to which a core dump will 
occur in the event of an abnormal termination. 

A shutdown/reboot API has been included in this draft. This 
includes options such as fast shutdown and graceful shutdown, 
and features such as rebooting with optional scripts. The config¬ 
uration-space-management API is intended to provide a 
portable method of traversing a systems configuration space, 
and for manipulating the data content of nodes in that configu¬ 
ration space. This API will provide a fault-management applica¬ 
tion access to underlying system configuration information and 
the means to direct reconfiguration of the system. The most 
recent changes in this area have been to move from a tree traver¬ 
sal to a directed graph. 

The working group has approved the current SRASS draft 4.0 
with changes to go out for ballot in December 1998; an extra 
meeting is planned at DISA in early December 1998 to verify 
that the latest changes have been incorporated correctly. If you 
are interested in helping support fault management (including 
serviceability and fault-tolerance aspects of systems), just get in 
touch with Helmut Roth at <hroth@nswc.navy.mil> or Dr. Arun 
Chandra at <achandra@vnet.ibm.com>. To subscribe to the SRASS 
mail list, send a message to <srass-request@pasc.org> and ask to be 
included. 
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Beginners 

A few years ago, there was a run of 
“introductory” UNIX books: Harley 
Hahns, Graham Glass’s, the new edition 
of the Nutshell book. Then there were 
the two “Dummies” books by John 
Levine and Margy Young (better than 
one might expect). McMullen’s UNIX 
User's Interactive Handbook puts these in 
the shade. It both permits the learner to 
access a training Web site, and also con¬ 
tains written exercises and on-screen 
lessons throughout. Chapter 8 on 
Regular Expressions and Chapter 9 on vi 
are very good indeed; if there’s any weak¬ 
ness, it’s the lack of discussion of shells. 
But perhaps that’s something we don’t 
want learners to know. McMullen has 
written a splendid work that takes the 
user from logging in and changing a 
password to customizing X. It’s a keeper. 

Spam 

Schwartz and Garfinkel {Stopping Spam) 
have produced a superb introduction to 
the problem of spam and the possible 
solutions. 

In nearly every form, spam is pervasive. 
My email today was just over 180 mes¬ 
sages; 42 of them were spam. I received 
11 pieces of US mail today; eight were 
unsolicited non-first class. Spam is intru¬ 
sive and offensive. But while catalogs, 
solicitations for donations, letters telling 
me that I “may already have won!”, and 
supermarket fliers may stuff my mailbox 
and burden my letter carrier, they do not 
cause service halts. In a period when ISPs 
have become more and more conscious 


of Quality of Service, the unwanted bur¬ 
dens of spam are truly onerous. 

Schwartz and Garfinkel have produced a 
well-written, entertaining book that con¬ 
tains a lot of information, including a 
solid chapter on blocking spam. (The 
section on sendmail 8.9 and the one on 
the getspam Perl script are particularly 
good.) 

Unfortunately, here and there Schwartz 
and Garfinkel wax excessively pious. 
Thus, on page 159 they credit a UDP 
(=Usenet Death Penalty) with causing a 
policy change at UUNET. Careful inquiry 
reveals that there was no change in 
UUNET’s policy, but that there was pub¬ 
lication of what the policy was, and that 
this wasn’t in response to the UDP, any¬ 
way. The assertions concerning both 
CompuServe (now part of AOL) and 
Netcom are suspect, too. 

The detailing of the “Green Card” 

(Canter and Siegel) scam and of the 
Spam King (Jeff Slayton) and his trans¬ 
gressions are extremely well done. I wish 
that the authors had discussed the Zilker 
suit (Austin, TX), which is not even men¬ 
tioned. However, the chapter on legal and 
legislative action is very good. 

All the URLs I tested worked. This is 
remarkable, as I frequently hit sites that 
have moved, vanished, or never were. 

Tcl/Tk 

Clif Flynt has produced a different sort 
of Tcl/Tk book {Tcl/Tk for Real 
Programmers ), and I hope it sells a lot of 
copies. Unlike a book on TCP/IP I have 
left unreviewed, which came with a CD- 
ROM for “Windows 3.1 and higher” and 
“16MB RAM and 2MB” memory, as well 
as PowerPoint, Flynt’s CD-ROM requires 
a 386 or better, “Windows 95 or NT, Mac 
OS7 or 8, or UNIX” and “4MB RAM or 
greater.” Let’s try to guess who’s aiming at 
a wider audience. 

Flynt writes lucidly and manages to soar 
above the level of Tel for Dummies, while 
shying from the depth found in the 
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O'Reilly or Addison-Wesley books. This 
really is for programmers and assumes at 
least some familiarity with C. I especially 
liked the sections on regular expressions 
and on exec. If you do scripting and/or 
gluing, this book will be a treat. 

C++ 

C++ supports classes, over-loaded func¬ 
tions, templates, modules, procedural 
programs, and more. Before I read 
Coplien's lucid and succinct essay (Multi- 
Paradigm Design for C++), I had never 
thought about multi-paradigm design. 
Programmers reading this book will see 
just how they can combine multiple par¬ 
adigms into their application develop¬ 
ment. Read together with Coplien’s other 
books on advanced C++ and on pattern 
languages, one can imagine the shape of 
C/C++ programming in the future. 

Y2K 

After nearly a decade of jaw-jaw-jaw, 
there appears to have been a general 
awakening to the problems of two-digit 
years over the past 18 months. 
Shakespeare's Year 2000 in a Nutshell is a 
welcome addition to the verbiage. Like so 
many of the O'Reilly books, it is narrowly 
focussed and thorough. Nearly a third of 
the book is composed of a COBOL quick 
reference. For those of you involved in 
the nitty gritty (as opposed to the poli¬ 
tics), this book will be a necessity. The 
first chapter should be valuable to those 
of you whose companies are unwilling to 
spend the money to rectify the problem. 


QoS 

Quality of Service (QoS) has become 
increasingly important. I reviewed 
Ferguson and Huston's slim book last 
June; I mentioned QoS with regard to 
spam, above. But it's necessary to under¬ 
stand that Internet efficiency is but the 
most obvious domain of QoS. In 1995, 
the Better Business Bureau received more 
complaints about computers than about 
car dealers. In terms of placement, com¬ 
puters (including software) has climbed 
from 20th place in 1994, to eighth in 
1995, to seventh in 1996 (the last year for 
which I have data). This is clearly a func¬ 
tion of the size of the potential usership 
as well as of the reckless distribution of 
insufficiently tested software. 

Kaner and Pels have written a book (Bad 
Software) designed for the irate and frus¬ 
trated software purchaser. I enjoyed the 
book and hope that it will empower folks 
to put a lot of pressure on the shoddy 
suppliers. 

Giroux and Ganti have turned out a very 
different sort of book (Quality of Service 
in ATM Networks). Where Kercheval's 
book barely mentions QoS and Ferguson 
and Huston devoted a chapter to it, 
Giroux and Ganti have lavished over 200 
pages on the topic. They've done a good 
job, too. As more and more commerce is 
transacted via the Internet, traffic man¬ 
agement in general will become vital; 
these two authors from Newbridge 
Networks provide a good example of 
what must be done. The only shortcom¬ 
ings I noted were the lack of discussion 
of dropped packets and of latency, which 
certainly influence QoS. 


Sys Admin 

For years, Nemeth et al. and Frisch have 
dominated the sys admin library. A new 
book by Lepage and Iarrera ( UNIX 
System Administrator's Bible) has come to 
(in some way) challenge this dominance. 
The best thing about Lepage and Iarrera's 
work is that it is simply presented. It also 
comes with a CD-ROM of FreeBSD 
2.2.5.1 admit that I wrote two of the 
chapters here (21 and 23) as well as part 
of the Introduction, so I’m not lavishing 
the space on the book it would otherwise 
deserve, but this is an entremely useful 
book. Well over a decade ago, I remarked 
to Lou Katz that I didn't want to be a sys¬ 
tem administrator. He remarked that if 
you ran UNIX, you’d be forced to be. 

This is still true, both for UNIX and for 
Linux. 

Cyberlife 

Books like David Bennahum’s Extra Life: 
Coming of Age in Cyberspace make me 
feel very old: I took FORTRAN a decade 
before he was born! But this brief biogra¬ 
phy of growing up in the period of Pong 
and Space Invaders really made me think 
about the history I have written about, as 
well as those born into it. Bennahum 
writes lightly and well, and I enjoyed the 
book tremendously. At times funny, at 
others touching, Bennahum has pro¬ 
duced a nontechnical, yet highly worth¬ 
while, memoir. 
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John Vacca 

Intranet Security 

Charles River Media, 1997. ISBN 1-886-80158-8. 

Pp. 506. CD included. $49.95. 

Reviewed by Terry Rooker 

<trooker@CapAccess.org> 

Intranets being a hot topic, a book on 
their security is worthwhile. Of course, 
intranets are just Internet technology 
used for a new purpose, so it seems that 
there shouldn’t be that much difference. 
As it turns out, using that technology for 
new purposes does change some of the 
assumptions, and you cannot simply 
apply security precautions developed for 
the Internet to the new application. 
Unfortunately, that is the weakness of 
this book. It rehashes all of the security 
techniques we’ve seen before and applies 
them to intranets. While the book is 
comprehensive in its coverage of the 
technology, the coverage is superficial 
and fails to explain how intranets may 
change some of the security require¬ 
ments. 

The book discusses user administration, 
some technical issues, virus detection and 
prevention, intrusion detection, and 
some legal issues involved in prosecuting 
those you discover. The level of detail is 
suitable for maybe first-line managers, 
and even some of them may find the 
detail a little lacking. Even though the 
discussion of certain issues includes some 
technical safeguards you can use, details 
on how to implement the safeguard are 
completely lacking. Furthermore, the 
approach of the author is to use off-the- 
shelf (OTS) software where available. The 
problem with using OTS software is that 
it causes additional concerns about its 
implementation and exactly what safe¬ 
guards it provides. In addition, you need 
extra analysis to ensure that the assembly 
of OTS products satisfies your security 
requirements. Unfortunately, this book 
does not discuss how to do that. So the 
repeated references to commercial prod¬ 


ucts makes the book appear more like an 
advertisement for those products - an 
image not helped by the enclosed CD, 
which includes demonstrations and ven¬ 
dor and product listings. 

This commercialism would be more 
acceptable if the book provided some 
insight into the new security concerns of 
intranets. Again it disappoints us, men¬ 
tioning that intranets raise new concerns 
but offering nothing to help us under¬ 
stand the problem. Yes, without an exter¬ 
nal network connection or a very limited 
connection, the main threat to an 
intranet is insiders. But what does that 
mean when we go to secure our network? 
This book provides few answers to that 
question. 

The superficial coverage of the technical 
issues makes Intranet Security of little 
value to network or system administra¬ 
tors. It is not worthwhile for readers try¬ 
ing to understand how intranets change 
the security issues. The comprehensive 
coverage of general network security 
issues is worthwhile, but then there are 
already many such books. The book does 
provide a good survey of some commer¬ 
cial products, so it would be worthwhile 
to someone looking for guidance to make 
a decision about purchasing commercial 
security products. 

Victor R. Volkman (ed.) 

C/C++ Treasure Chest 

R&D Books. ISBN 0-87930-514-2. Pp. 224. $39.95. 
Includes CD. 

Reviewed by Clif Flynt 

<clif@cflynt.com> 

One of the features that distinguishes 
experienced programmers from newbies 
is the bag of tricks that old-timers carry 
around in their heads (or on a floppy 
disk). These tried-and-true code snippets 
let the old pros go straight to solving a 
problem, instead of figuring out how to 


handle the details. We call this code 
reuse. 

The biggest hurdle to code reusability is 
finding the code you want to use. I sus¬ 
pect that all of us have spent days writing 
some marvelously clever code, only to 
have a friend say, “Why didn’t you just 
steal the code from the foo package?” 

For the past 15 years, the C/C++ Users 
Group has collected source code and 
made it available for other programmers 
to examine, use, and steal ideas from. The 
trick has been finding the package that 
has what you need now. C/C++ Treasure 
Chesty compiled by Victor Volkman, 
makes a big dent in solving this problem. 
The book contains brief descriptions of 
literally hundreds of C and C++ pro¬ 
grams, libraries, and documentation, 
indexed by keyword, type of functionali¬ 
ty, OS/CPU, and title. 

Along with the book is a CD-ROM con¬ 
taining the complete C Users Group code 
distributions, with HTML indexes to 
direct users to the package they need. 
These indexes and HTML pages make the 
difference between 400 packages of data 
and 400 packages of information. 

This book will pay for itself if you find a 
single package that saves you an hour. My 
bet is that you’ll find more than one 
package, be it the genetic-algorithm and 
neural-network toolkits, the OS-indepen¬ 
dent screen editors, the JPEG library, 

SQL package, linked-list library, the MS- 
DOS TSR tools, or other packages. The 
range of code available is astounding, 
and, most important, you can find the 
code you need. 
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Jody Leber 

NT Backup and Restore 

O’Reilly & Associates, 1998. ISBN 1-56592-272-7. 
Pp. 320. $29.95 

Reviewed by Steve Hanson 

<hanson_steve@htc.honeywell.com> 

Although this book is aimed at the NT 
backup administrator, it is primarily a 
general guide to backup strategy. It gives 
a good overview of backups - why they 
are important, how to plan and scale 
them, and how to select a tool for back¬ 
up. Most of this material is as applicable 
to a UNIX network as it is to an NT net¬ 
work. In fact, there is only one chapter 
devoted specifically to NT backup issues 
and the reasons why NT is particularly 
challenging (proper backup of the reg¬ 
istry and WINS database, etc.). I was dis¬ 
appointed that these issues were not cov¬ 
ered in more depth, since doing NT 
backup is one of the issues I’m dealing 
with at work. The book didn’t help me 
much with my issues, such as backing up 
open NT files. 

The book does have many strengths, 
however. Several tables help the admin to 
understand how many tape devices and 


how much network bandwidth will be 
needed to do backups within a given 
time window. If your site is anything like 
mine, getting backups done during an 
off-hour window is an ever-increasing 
challenge as disk space grows. The charts 
help to determine whether that new file 
server is going to throw you over the 
edge. Likewise the book gives attention to 
reviews of a number of commercial NT 
backup products as well as those that are 
freely available with the OS. And a fea¬ 
ture-comparison chart for commercial 
backup products is nearly as useful to the 
UNIX admin as to the NT administrator. 

This is the only book on backup that I 
know of, NT or UNIX. This is surprising 
- we’d all agree that backup is one of the 
most important things we do, yet there 
are very few places to learn how to do it 
well. Many UNIX and NT admins are 
finding that those commercial backup 
products are starting to look good as the 
size of our backup problem increases. 
This book is a helpful guide to backup in 
general and how to make sense of the 
commercial alternatives available to the 
NT world. Yet I wished that it went into 
more detail on some of the pitfalls and 
how to avoid them. Many topics (such as 


different backup schedules) are covered, 
but not really in enough detail to be 
helpful to the professional who has been 
doing this for some time. I’d recommend 
this book to the beginning administrator, 
and particularly to anyone who is consid¬ 
ering the purchase of a commercial back¬ 
up system for NT. It is even fairly useful 
to the UNIX administrator, as most of 
the topics covered apply to both environ¬ 
ments. However, the old hand at doing 
backups isn’t likely to gain much from 
the book, with the possible exception of 
the capacity charts. Even those may not 
be very useful to anyone with reasonable 
math facilities, as they are primarily divi¬ 
sion tables in disguise. 


Thanks to Our 1998 Reviewers! 

Here is our honor roll of book reviewers for the past year! Join the crowd: review a book for us! 

William S. Annis, Reginald Beardsley, Nick Christenson, Clifton Flynt, Andrew Hume, Rob Jenson, Chris Kottaridis, Daniel Lazenby, 
George W. Leach, Bruce O’Neel, Terry Rooker, Carolyn J. Sienkiewicz, Kartik Subbarao, and Rick Umali. 

Again, thank you! 
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USENIX news 


In Memoriam: 

John Lions 

by Peter H. Salus 

John Lions, the principal instigator of 
UNIX in Australia, founder of the 
AUUG, and author of the Commentary 
on V6 (which Ken Thompson called “the 
best book on how an operating system 
works”), died on Saturday, 5 December 
1998. 

John had been quite ill for nearly five 
years. 

John earned degrees from Sydney 
University and Oxford. He worked for 
Burroughs in Canada until he went to the 
University of New South Wales in 1972. 
He retired in July 1995. 

In 1975-76, while teaching a course in 
operating systems, John wrote a com¬ 
mentary on the source code for his stu¬ 
dents. The booklets (in bright red and 
orange covers) were announced in UNIX 
NEWS (the ancestor of ;login:) and the 
UKUUG newsletter, only to be sup¬ 
pressed by AT&T lawyers. In 1993, 
Dennis Ritchie and I attempted to obtain 
permission to publish these often-xerox¬ 
ed pamphlets, but only succeeded in 
1996, when the rights were sold to SCO; 
John’s masterpiece is finally available. It 
was reviewed by Jaap Akkerhuis in 
Matrix News 704 (April 1997). The 
Commentary and Source Code is avail¬ 
able as ISBN 1-57398-013-7; the 
Japanese translation is ISBN 4-7561- 
1844-5. 

Greg Rose (Qualcomm, Australia), who 
was one of John’s students and is now 
Vice President of USENIX, wrote: 

“While at the University of New South 
Wales, John introduced a course in 
Operating Systems, and decided to study 
the UNIX operating system. One of his 
motivations in doing this was to intro¬ 
duce the students to code which was well 
written by other people - at the time this 


was not a common practice, although it 
is now well accepted - and which imple¬ 
mented a very significant system. In the 
course of developing notes for this, he 
wrote an annotation of the source code 
of UNIX, and produced a pair of books 
(recently republished and translated) 
including the source code itself. This was 
a remarkable achievement, and demon¬ 
strated the clarity of thought of which he 
was capable. The books were not avail¬ 
able for general distribution at the time, 
but were probably the most successful 
illegally copied books ever; there are 
numerous reports of‘5th generation 
photocopies’. 

“I remember an incident when I was a 
student of John’s, simultaneously helping 
to run the PDP-11 computer in the then 
Department of Computer Science, where 
the computer was often ‘locking up’ 
under high load. John took home listings 
of the current source code, and returned 
the next morning with details of two race 
conditions and a potential deadlock in 
the UNIX kernel which might have 
explained the problem, and indeed when 
they were fixed the problem went away. 

“John pulled together a group of people 
interested in UNIX, and when it was 
later formalized as the Australian UNIX 
Users’ Group became the founding presi¬ 
dent of the organization. 

“Within the University of New South 
Wales there was a battle over centraliza¬ 
tion versus distribution of computing 
resources, which indirectly had a major 
effect on the autonomy of the 
Department of Computer Science. John’s 
battle to have UNIX accepted as a vehi¬ 
cle for teaching, and later as the subject 
of teaching, instrumentally led to the 
increasing importance and independence 
of the department. Further, the existence 
of a centre at UNSW helped the forma¬ 
tion of similar groups at Melbourne and 
Sydney Universities.” 

John Lions was a fine teacher and a good 
friend. I am certain that all members of 
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the Association join me in expressing 
heartfelt condolences to his widow, 
Marianne. 

Changes 

Lately I have been in a contemplative 

by Andrew Hume \ 

President, USENIX Board of Directors 

<andrew@research.att.com> , 


mood. There are several good reasons 
why. I have started mentoring (via email) 
a female undergraduate at a university in 
the Midwest. The project I have spent the 
last 3 1/2 years on is increasingly under 
its own management and direction. In 
April, my wife is expecting our first chil¬ 
dren (twins!). And, worst of all, I am get¬ 
ting really cranky about the software I 
use every day. 

From 1983 to 1996 I worked in computer 
science research at Bell Labs, beside some 
of the best in our field (Ritchie, 
Thompson, Kernighan, Mcllroy, Pike,...). 
Even better, I got to use their software! 
Eighth, ninth and tenth editions of 
Research UNIX, and the various editions 
of Plan 9. Best of all, we had Dave 
Presotto riding herd on the two greatest 
fiascos of modern computer systems, 
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mailers and networking. During this time 
I was quite productive in my work, large¬ 
ly due to the quality and smoothness of 
my computing environment, especially 
the effective pervasive user interface and 
window system done by Rob Pike. 

Strangely, I was often pitied by outsiders 
as stagnating in my own insular backwa¬ 
ter. I was missing out on the software rev¬ 
olution, the brave new world of the PC 
with its stunning, fabulous diversity of 
really keen products to help me do my 
work and increase my productivity. When 
I moved from Bell Labs to AT&T Labs, I 
undertook to experience the mass market 
full on! I tried to live on the frontier, 
using Windows 95 and X. Furthermore, 
my projects involved production systems, 
so 1 got to see and use real software (not 
that fake stuff we had in Research). After 
three years, I have learned some lessons 
and thought I'd pass them along. Of 
course, these are just one persons opin¬ 
ions; I would expect your mileage to vary. 

1. As far as I can tell, the only useful out¬ 
come of the PC business is high-per¬ 
formance commodity hardware. The 
software sucks. Even at its worst 
moments, Plan 9 was more reliable 
than Windows 95. Of course, it worked 
okay (meaning it only crashed every 
week or two) if you didn't do much 
networking or open many windows or 
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install any new software. But this 
sounds like giving up to me. 

2. But what about the fabulous market¬ 
place of software to do my bidding? 
Well, I tried that. I wanted some 
JPEG/MPEG viewers. So I followed the 
new order and went to <www.tucows.com> 
and downloaded every freeware/share¬ 
ware viewer they had. Of the fifteen I 
tried, five destroyed my Windows soft¬ 
ware (complete reload required), and 
eight coredumped or caused the system 
to crash. But two actually worked! One 
of these had one of the worst user 
interfaces Eve seen in some time; one 
was actually usable. 

3. So I am giving up on the whole 
Windows thing. I am switching to 
Linux or a BSD system; it should be 
much better, and I can't imagine it 
being worse. 

4. Perhaps more important, I think I am 
shifting to the open source camp. 
Perhaps 1 was unlucky, but in my cur¬ 
rent project we used five distinct soft¬ 
ware products on the production 
machine (other than what we devel¬ 
oped). Three were commercial offer¬ 
ings (the system software, a backup 
suite, and a file transfer suite), and two 
were supported by folks in AT&T Labs 
- Research (sorting and searching pro- 
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grams). After fifteen months, the story 
was: 

■ Searching: 0 bugs, high performance 

■ Sorting: 5 bugs, two months to fix 

■ Backup: 10+ bugs, several tapes’ worth 
of data lost, mediocre performance, 1 
fix delivered during severity-1 mode 

■ File transfer: 5 serious bugs, 3 fixed by 
patches within a couple of months, 2 
still unresolved after a year 

■ System software: 10+ serious bugs, all 
but 1 fixed within 2-5 months 

I seem to have come full circle back to 
where I started in 1975 - running UNIX 
and software developed by a worldwide 
collaborative community. And for pretty 
much the original reasons: its a very 
effective way to use the hardware, and the 
software (and software environment) is 
simply better. (Of course, I’m still hoping 
for another release of Plan 9!) Although 
some part of me is depressed at the 
apparent lack of progress in 24 years, the 
rest of me is excited at getting back to an 
environment where I am much more 
effective. 


Board Meeting 
Summary 


by Ellie Young 

Executive Director 
<ellie@usenix.org> 


Here is a summary of the actions taken at 
the regular meeting of the USENIX 
Board of Directors held on November 5- 
6, 1998, in Berkeley, CA. 

Attendance: USENIX Board: Geer, 
Honeyman, Hume, Parseghian, 

Pomeranz, Rose, and Zwicky. USENIX 
Staff/Guests: Young, Berkowitz, Barnett, 
DesHarnais, Klein, Deno, Long, 
Appelman, Johnson, Miller, Stoughton, 
and Suski. 

Proposals for Funding 

USACO. The proposal to continue fund¬ 
ing the USA Computing Olympiad 
(which selects and trains the US team for 
the International Olympiad in 
Informatics) was approved. 

Software Patent Institute. A proposal 
from the Software Patent Institute for a 
grant of $55,000 to continue to expand 
and improve the SPI database of software 
technologies was approved. This database 
is presently accessible to the public, with 


the goal of helping patent applicants and 
the US Patent & Trade Office issue vali¬ 
dated patents in the software field. 

Incident Cost Analysis & Modeling 
Project Frequency Study. A request from 
the University of Michigan for funding 
support of $64,510 for this project was 
approved. The project is designed to pro¬ 
vide system administrators with addi¬ 
tional information regarding which IT 
related risks are of highest priority to 
address, about factors relating to occur¬ 
rence, costs, and how best to manage cer¬ 
tain risks, and how to cost-effectively 
manage incidents. The project also pro¬ 
vides a valuable and active learning envi¬ 
ronment for graduate students wishing to 
explore the economic as well as technical 
aspects of IT related events. 

USENIX Scholastic Committee. An addi¬ 
tional $20,000 was allocated to fund 
research grants and scholarships for the 
December 1998 round of applications. 

Standards. Stoughton’s proposal for a 
moderate increase in our involvement, 
covering the new POSIX revision proj¬ 
ects, membership in The Open Group, 
and attending meetings of the TOG Base 
Working Group was approved. Hall and 
Stoughton will look into the possibility of 
funding members of the Linux commu¬ 
nity to attend meetings. 
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Berkeley Foundation for Opportunity in 
Information Technology. It was agreed to 
become a founding donor of this project 
with a donation of $10,000. BFOIT has 
been recently formed to assist women 
and underrepresented minority students 
enter the IT industry. Assistance will 
include providing financial aid for stu¬ 
dents, create relationships with local IT 
corporations for jobs and internships, 
and other support as needed for students 
doing IT coursework in CS and electrical 
engineering. 

Draft Budget 1999 

The assumptions behind the first draft 
budget for 1999 were discussed. It was 
decided that we should continue to bud¬ 
get conservatively for conference atten¬ 
dance, especially since this is an area that 
is vulnerable to economic downturns; 
that we increase the marketing budget for 
the Security symposium; that we consider 
increasing fees and dues to handle the 
increase in the number of events we are 
sponsoring (going from 8 to 12 in 1999), 
the rising costs of holding conferences, 
and the increased costs of serving the 
membership. 

A marketing advisory committee of the 
Board was formed to look into issues 
such as brand recognition, how we mea¬ 
sure and correlate the results of our mar¬ 
keting efforts, and how we can increase 
attendance at some of our smaller or 
newer conferences. 

Membership Dues 

After discussion about the increased costs 
of providing services to a greater number 
of members (while dues have not been 
raised since 1994), it was agreed to set 
fees as follows: 



1999 

2000 

Individual 

$80 

$95 

Supporting 

$1000 

$1000 


$2500 

$2500 

Educational 

$200 

$200 

Corporate 

$400 

$400 

Affiliates 

$75 

$90 


Conference Registration Fees 

The staff’s report revealed that the costs 
for putting on conferences have skyrock¬ 
eted in the past couple of years, while 
USENIX registration fees continue to be 
among the lowest. It was agreed to raise 
fees for the next two years as follows: 


Tutorials 

1999 

$395 one day 
$690 two days 
$985 three days 


2000 

$445 one day 
$740 two days 
$1035 three days 


Three Days of Tech Sessions 


1999 2000 

$400 $435 


Two Days of Tech Sessions 

1999 2000 

$360 $410 


It was also decided the conference regis¬ 
tration fees for students will remain the 
same ($75), and the non-member confer¬ 
ence fees will be the conference member 
rate plus the cost of an individual mem¬ 
bership. 


Electronic Voting 

It was agreed that Young will look into 
enabling electronic voting for the next 
USENIX Board elections, especially 
exploring what the auditors’ recommen¬ 
dations would be regarding tabulating 
votes electronically. 


Privacy Statement 

The Board voted to adopt a USENIX pol¬ 
icy statement and to publish it in ;login: 
(see page 82) and on our Web site. 

Staffing 

Young reported that there had been a lot 
of staff turnover recently. She has hired a 
new publications director, Jane-Ellen 
Long, to replace Eileen Cohen, who was 
retiring from full-time work; she has 
hired a Deputy Executive Director, Gale 
Berkowitz, to handle operations, SAGE, 
finances and other projects; two other 
admin positions for member services and 
publications/ 


Web have been filled (by Cami Edwards 
and Jennifer Radtke), and the conference 
office has hired a new assistant, Vanessa 
Fonseca. It is anticipated that additional 
personnel for marketing and the execu¬ 
tive office will be needed in 1999. 

Embedded Systems Conferences 

It was agreed to co-sponsor this confer¬ 
ence with the MIT Media Lab with 
Michael Hawley and Dan Geer as co¬ 
chairs. 

In Memoriam 

It was agreed to publish tributes to the 
recent deaths of Jon Postel and John 
Lions. 

BOD Next Meeting 

The next meeting will be held February 
21 in New Orleans, LA. 

20 Years Ago in UNIX 

by Peter H. Salus 

<peter@pedant.com> 

__ j 

Note another change in title! ;login: 

(a.k.a. UNIX NEWS) did not reappear 
until 1980.1 wrote last issue that I’d talk 
about the Association and its activities in 
1999. But I’d like to say a bit about 
UNIX, too. 

1978 saw the appearance of Version 7, the 
first BSD tape (containing Pascal and 
“the ex editor”), the Ritchie-Johnson port 
to the Interdata 8, the Wollongong port 
to the Interdata 7, the port to 32V by 
Charlie Roberts and a group at Holmdel, 
the publication of uucp, and the first 
meeting of the NLUUG - and, of course, 
the publication of The C Programming 
Language and the “blue” BSTf. 

It may be worthwhile to itemize some of 
the things found for the first time in V7: 
intro (introduction to commands), adb 
(debugger), at, awk, calendar, cb (the C 
beautifier), cd, cu, derof f, expr, f77 
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